Skip to main content

software catalog

Updated Sep 24, 2026

Quickstart

← Docs index

This walkthrough serves Qwen3-4B with a single docker run.

Complete the Prerequisites once before you start. If you plan to serve a larger model, check memory sizing first. For replicas and secrets, use Kubernetes instead.

Tip: Want to serve other models? Open the Intel® Software Catalog, find your model, and copy the docker run from the Deployment tab. If your model is not in the catalog, use the base image — Deploy an image that isn't in the catalog.

1. Start the server — leave this terminal open

The first start downloads the model and can take several minutes. Leave this terminal open; this process is the server. Open a new terminal for step 2, and continue when the model is alive.

bash 
docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0
  • --cap-add SYS_NICE — lets the engine pin memory to the CPU socket it is using.
  • --shm-size=2g — Docker gives a container only 64 MiB of shared memory by default. The engine needs more than that to start, so without this flag the container exits. 2g is enough.

This command downloads the model on every launch. To cache the weights instead, see Model caching.

You can also watch docker ps: STATUS goes from (health: starting) to (healthy).

If the model is gated

Gated models need a Hugging Face token. Llama and Gemma are gated; Qwen3-4B is not. Create a Read token and accept the license: Prerequisites, Hugging Face. Then copy the docker run from that model's Deployment tab in the Intel® Software Catalog. The example below is Llama 3.2 3B Instruct — replace <your-token> with the token you created.

bash 
docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e HF_TOKEN=<your-token> \
  intel/inference-xeon-meta-llama-llama-3.2-3b-instruct:0.1.0

A token on the command line is saved in shell history. Security shows how to avoid that. This command also downloads the model on every launch; Model caching explains how to skip that wait.

2. New terminal — wait until it is alive, then chat

The first start can take several minutes while the weights download. Poll /health until you see It's alive, then send a Hello request. This image serves Qwen/Qwen3-4B, so you can use that id as-is — you do not need to copy one from /v1/models.

bash 
curl -sf http://localhost:8000/health && echo "It's alive — the model is listening. Say hello."
bash 
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-4B",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

If /health never returns 200, turn up the logs and see Troubleshooting. Later starts skip the download once a cache mount is in place — Model caching.

3. Call it from Python

Point the OpenAI Python SDK at http://localhost:8000/v1. The server does not use api_key, but most clients still expect a non-empty string.

Save the following as hello.py and run python3 hello.py:

python 
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="Qwen/Qwen3-4B",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

4. Call it from LangChain, LlamaIndex, LiteLLM, or Haystack

The container is an OpenAI-compatible endpoint. There is no Intel® client to install. Any framework that speaks the OpenAI API can call it — point it at http://localhost:8000/v1. The four below are the most common; they are examples, not a closed list.

FrameworkWhere to set the base URL
LangChainChatOpenAI(base_url="http://localhost:8000/v1", ...)
LlamaIndexOpenAILike(api_base="http://localhost:8000/v1", ...)
LiteLLMopenai/ model prefix + api_base
HaystackOpenAIChatGenerator(api_base_url="http://localhost:8000/v1", ...)

Full copy-paste examples, including api_key, are in Framework integrations. Keep the docker run terminal open; stopping the container also stops the endpoint.

5. Stream the reply

This is the same Hello request, with "stream": true. -N tells curl not to buffer. The rest of the pipeline prints only the words as they arrive, not a JSON object per token.

bash 
curl -N -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-4B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": true
}' | awk -F '"content":"' 'NF>1 { split($2, a, "\""); printf "%s", a[1]; fflush() } END { print "" }'

Qwen3-4B may think out loud first; the greeting follows. That is the model, not a stalled stream.

The wire format (one data: JSON line per token) and the OpenAI SDK streaming loop are in the API.

Check the configuration without serving (optional)

dry-run prints the config and the vLLM command it would run, without loading weights:

bash 
docker run --rm \
  --cap-add SYS_NICE \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0 \
  dry-run --format json

--format can be json or yaml. For the other commands (serve, list-profiles, download-to-cache, detect-hardware), see the CLI.

Congratulations — you have the model running

The curl and code snippets above were only to confirm the endpoint. In a real app, keep using the same OpenAI-compatible client you use today.

From here you can:

PreviousPrerequisitesNextModel Catalog