← Docs index

What you can customize

HF_TOKEN and engine-native variables such as VLLM_DO_NOT_TRACK are not read by this runtime directly (INFERENCE_TRUST_REMOTE_CODE is — it is read here and translated into an engine flag). A value you pass with -e always wins.

A catalog model image starts with no extra variables. Use this page when you need a Hugging Face token, a different port, or an override. Skip HF_TOKEN unless the listing's docker run includes it.

Pass each setting as -e NAME=value before the image name on docker run, or under env: on Kubernetes. The container reads them once at startup — change a value, restart the container or pod. CLI flags such as --format only change printed output.

Each command that uses "$IMAGE" is a catalog model image, not intel/inference-xeon-base. Copy it from the last argument of that listing's docker run — for example intel/inference-xeon-qwen-qwen3-4b:0.1.0. You only need a Hugging Face id (INFERENCE_MODEL_ID) on the base image. Named walkthrough: Quickstart. Commands: CLI.

Check a value without serving (dry-run does not load weights or bind port 8000):

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm --cap-add SYS_NICE -e INFERENCE_PORT=9000 "$IMAGE" dry-run --format json

Passing a variable

Put -e before the image name. You can stack as many as you need.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_PORT=9000 \
  "$IMAGE" \
  dry-run --format json

Wrong: docker run IMAGE -e ... — Docker treats that as a container command, not an env var.

Checking that a value took effect

Run dry-run --format json on the same model image you would serve, with the same -e values. It prints the resolved config. It does not load weights and does not bind port 8000, so it is safe while another container is already serving.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_PORT=9000 \
  "$IMAGE" \
  dry-run --format json

Look in the JSON:

Then copy the same names into the Deployment env: list (Kubernetes). Do not leave a checked value only on a local docker run -p.

Settings you may need

Which model, its token, the port, the cache directory, and the log level.

INFERENCE_MODEL_ID — which model (base image only)

You do not copy a model id from the catalog. A model image already has it — paste the listing's docker run and skip this variable.

When: only if you are using intel/inference-xeon-base:0.1.0 because your model is not in the catalog. If Intel® has a validated configuration for that Hugging Face id, the container applies it. If not, the model still serves, with the standard runtime configuration.

# Hugging Face model ID, for example: Org/Model-Name
export MODEL_ID="Org/Model-Name"

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_MODEL_ID="$MODEL_ID" \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  intel/inference-xeon-base:0.1.0

Without it, the base image fails with INFERENCE_MODEL_ID is required. Same name under env: on Kubernetes. Check with dry-run --format json and look at "model_id".

The chat API still needs a "model" field. That string is the Hugging Face id the container is serving. On the Quickstart it is Qwen/Qwen3-4B and you type it. For any other image, ask the server and save it:

export MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)

Use "$MODEL_ID" as "model" in the request. grep / cut are already on Linux. Details: API reference.

HF_TOKEN — gated Hugging Face models

When: the listing's docker run includes -e HF_TOKEN. How to serve a gated model is already in Quickstart — If the model is gated. Create the token and accept the license: Prerequisites. Putting the token on the command line writes it to Linux shell history — Security.

INFERENCE_PORT — listen on a port other than 8000

When: host port 8000 is already in use. Set the env var and map the same port with -p.

Check first (dry-run does not bind the port):

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_PORT=9000 \
  "$IMAGE" \
  dry-run --format json

Expect in "command_script": --port 9000.

Then serve:

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm -p 9000:9000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_PORT=9000 \
  "$IMAGE"

Call http://localhost:9000/health and http://localhost:9000/v1/.... -p 9000:8000 without -e INFERENCE_PORT=9000 is wrong: the process still listens on 8000 inside the container.

The image's EXPOSE is fixed at 8000 — Docker cannot make it dynamic — so a non-default port is never published implicitly (docker run -P publishes the image's exposed port (8000) only). Always pass -p <port>:<port>. The image's HEALTHCHECK does read INFERENCE_PORT, so docker ps reports healthy on the port you chose. The runtime logs a warning at startup whenever INFERENCE_PORT is not 8000, as a reminder that the publish/probe settings must follow.

On Kubernetes, set INFERENCE_PORT, containerPort, Service port / targetPort, and probes to the same number.

INFERENCE_CACHE_PATH — cache directory inside the container

When: you mount the cache somewhere other than /workspace/model-cache. The -v right-hand side and this variable must be the same path. See Caching.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_CACHE_PATH=/alt-cache \
  -e INFERENCE_LOG_LEVEL=DEBUG \
  "$IMAGE" \
  dry-run

DEBUG logs should mention /alt-cache. To actually use it:

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_CACHE_PATH=/alt-cache \
  -v /data/cache:/alt-cache \
  "$IMAGE"

INFERENCE_LOG_LEVEL — more (or fewer) framework logs

When: startup is failing or you want to see hardware detection and which built-in config was picked. This does not turn up vLLM itself — that is INFERENCE_LOG_LEVEL_ROOT. Details: Logging.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_LOG_LEVEL=DEBUG \
  "$IMAGE" \
  dry-run --format json

Expect extra lines about hardware detection and Selected profile. WARNING is quieter.

Advanced: overriding what the image decided

These four override what the container decided at startup. Each one moves you off the Intel®-validated configuration for that model. Check the result with dry-run --format json before you rely on it.

INFERENCE_ENGINE_ARGS — extra vLLM flags

When: you need a one-off engine option on top of the built-in config (for example a shorter context). Value must be a JSON object. Malformed JSON is ignored with a warning. A flag vLLM rejects fails at startup. Check dry-run either way.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_ENGINE_ARGS='{"max-model-len": 4096}' \
  "$IMAGE" \
  dry-run --format json

Expect in "command_script": --max-model-len 4096.

On the CPU backend, gpu-memory-utilization is passed the same way — it sets the fraction of a NUMA node's memory vLLM reserves (default 0.92). Lower it when startup fails with Available memory on node 0 ... less than desired CPU memory utilization:

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_ENGINE_ARGS='{"gpu-memory-utilization": 0.5}' \
  "$IMAGE" \
  dry-run --format json

Expect in "command_script": --gpu-memory-utilization 0.5. See Troubleshooting.

INFERENCE_TRUST_REMOTE_CODE — allow or forbid model-provided code

When: a model you name yourself on the base image ships its own Python (Hugging Face calls this "remote code"), or you want to forbid that outright as a matter of policy.

Some models are not implemented by the engine itself: loading them runs Python published in the model's Hugging Face repository, inside your container, before the API ever answers. Catalog model images already carry the right setting — you do not need this variable for them.

Leave it unset and the container decides: a catalog model image uses its own setting, and for a model you name yourself the container inspects the model first and enables execution only if the model declares its own Python, logging why. Set the variable to take that decision yourself — which also skips the lookup:

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

# forbid it, whatever the image says - the hardened production setting
docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_TRUST_REMOTE_CODE=false \
  "$IMAGE" \
  dry-run --format json

INFERENCE_PROFILE_ID — pin a built-in config

When: you want a specific row from list-profiles instead of auto-pick. Leave unset for the default ([primary]). You do not author profiles.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm --cap-add SYS_NICE "$IMAGE" list-profiles
docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_PROFILE_ID=vllm-xeon-bf16 \
  "$IMAGE" \
  dry-run --format json

Expect: "profile_id": "vllm-xeon-bf16".

INFERENCE_ACCELERATOR_COUNT / INFERENCE_ACCELERATOR_MODEL — override detection

When: the container sees the wrong core count (cgroup) or you are dry-running on a different box than production.

export IMAGE=<intel/inference-...:tag>  # e.g. intel/inference-xeon-qwen-qwen3-4b:0.1.0

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_ACCELERATOR_COUNT=8 \
  "$IMAGE" \
  dry-run --format json

Expect: "accelerator_count": 8. Unrecognized INFERENCE_ACCELERATOR_MODEL values fall back to auto-detection with a warning.

Leave these alone unless you have a reason

Related pages

• CLI — dry-run — the command that shows whether a value took effect
• Deploy on Kubernetes — the same names under env:
• Logging — the two log-level variables in detail
• Model caching — INFERENCE_CACHE_PATH in context
• Security — passing HF_TOKEN without leaking it
• Supported features — what INFERENCE_ENGINE_ARGS unlocks, and the caveats