software catalog

Updated Sep 1, 2026

Quickstart

← Docs index

Serve Qwen3-4B with one docker run.

Prerequisites — do these once: Prerequisites. Larger models: memory sizing. For replicas and secrets: Kubernetes.

Tip: Want to serve other models? Open the Intel® Software Catalog, find your model, and copy the docker run from the Deployment tab. If your model is not in the catalog, use the base image — Deploy an image that isn't in the catalog.

1. Start the server — leave this terminal open

The first start downloads the model and can take several minutes. Leave this terminal open — it is the server. Open a new terminal for step 2. Continue when you see that the model is alive.

bash 
docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0
  • --cap-add SYS_NICE — lets the engine pin memory to the CPU socket it is using.
  • --shm-size=2g — Docker gives a container only 64 MiB of shared memory by default. The engine needs more than that to start, so without this flag the container exits. 2g is enough.
  • -v is model caching: weights stay on your machine so later starts skip the download. Mount at /workspace/model-cache/huggingface — the container is not root (Security).

You can also watch docker ps: STATUS goes from (health: starting) to (healthy).

If the model is gated

Gated models need a Hugging Face token. Llama and Gemma are gated; Qwen3-4B is not. Create a Read token and accept the license: Prerequisites, Hugging Face. Then copy the docker run from that model's Deployment tab in the Intel® Software Catalog. This is Llama 3.2 3B Instruct — replace <your-token> with the token you created.

bash 
docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e HF_TOKEN=<your-token> \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  intel/inference-xeon-meta-llama-llama-3.2-3b-instruct:0.1.0

A token on the command line lands in shell history — Security shows how to avoid that.

2. New terminal — wait until it is alive, then chat

The first start can take several minutes while weights download. Run health until you see It's alive, then send Hello. This image is Qwen/Qwen3-4B — you do not copy an id from /v1/models.

bash 
curl -sf http://localhost:8000/health && echo "It's alive — the model is listening. Say hello."
bash 
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-4B",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

If /health never returns 200, turn up the logs — Troubleshooting. Later starts skip the download when the cache mount is in place (Model caching).

3. Call it from Python

Point the OpenAI Python SDK at http://localhost:8000/v1. api_key is unused by the server; most clients still want a non-empty string.

Save as hello.py and run python3 hello.py:

python 
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="Qwen/Qwen3-4B",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

4. Call it from LangChain, LlamaIndex, LiteLLM, or Haystack

The container is an OpenAI-compatible endpoint. There is no Intel® client to install. Any framework that speaks the OpenAI API can call it — point it at http://localhost:8000/v1. These four are the most common; they are a reference, not a closed list.

FrameworkWhere to set the base URL
LangChainChatOpenAI(base_url="http://localhost:8000/v1", ...)
LlamaIndexOpenAILike(api_base="http://localhost:8000/v1", ...)
LiteLLMopenai/ model prefix + api_base
HaystackOpenAIChatGenerator(api_base_url="http://localhost:8000/v1", ...)

Copy-paste for each, including api_key: Framework integrations. Keep the docker run terminal open; stopping the container takes the endpoint with it.

5. Stream the reply

Same Hello request, with "stream": true. -N tells curl not to buffer. The rest of the line prints only the words, as they arrive — not a JSON object per token.

bash 
curl -N -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-4B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": true
}' | awk -F '"content":"' 'NF>1 { split($2, a, "\""); printf "%s", a[1]; fflush() } END { print "" }'

Qwen3-4B may think out loud first; the greeting follows. That is the model, not a stalled stream.

The wire format (one data: JSON line per token) and the OpenAI SDK loop: API.

Check the configuration without serving (optional)

dry-run prints the config and vLLM command without loading weights:

bash 
docker run --rm \
  --cap-add SYS_NICE \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0 \
  dry-run --format json

Flags: CLI.

That is enough to build against

The model is serving. curl and the snippets above were only to prove it. In a real app you keep using the same OpenAI-compatible client as today.

PreviousPrerequisitesNextModel Catalog