← Docs index

Errors you can hit with Intel® Inference Microservices, indexed by the message you actually see. Most failures happen at startup, so reproduce them with dry-run, list-profiles, or detect-hardware — none of those load model weights or bind port 8000 — before you change anything in production.

Each command that needs an image exports IMAGE — the last argument of the catalog listing's docker run. "$MODEL_ID" is the Hugging Face id on the base image only — Environment variables. A named walkthrough is in Quickstart.

Find your error

contains custom code which must be executed

The model publishes its own Python in its Hugging Face repository, and the container did not enable executing it. Normally the container detects that and enables it for you, so seeing this error means one of three things:

• Someone set INFERENCE_TRUST_REMOTE_CODE=false — a deliberate denial, which nothing else can override. That is the intended behavior, not a bug.
• The pre-flight check could not reach the Hub, or the repository is gated and no HF_TOKEN was set. The log says so: Could not determine whether ... requires remote code execution.
• The model needs it in a way the check cannot see from repository metadata.

Confirm which way it resolved:

export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE -e INFERENCE_MODEL_ID="$MODEL_ID" "$IMAGE" dry-run --format json

Look at "trust_remote_code". If it is false and you have decided to trust whoever publishes that repository:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_MODEL_ID="$MODEL_ID" \
  -e INFERENCE_TRUST_REMOTE_CODE=true \
  "$IMAGE"

Read Security first — that code runs inside your container, with access to its environment, before the API answers. Where the model is in the Intel® Software Catalog, use its model image instead of the base image: the id is fixed at build time and the setting ships with it.

No such command 'list'

The command is a single token: list-profiles. A space makes the CLI look for a command named list, which does not exist.

export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE "$IMAGE" list-profiles

See every command:

export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE "$IMAGE" --help

The same applies to dry-run, download-to-cache, and detect-hardware.

pull access denied / repository does not exist

Docker went looking for that name in a registry and did not find it. Check what you already have locally before anything else:

docker images | grep inference

If the image is listed under a different name, run that name.

To Docker, inference-xeon-org-model:0.1.0 (a local tag) and intel/inference-xeon-org-model:0.1.0 (a Hub name) are two unrelated images. The first is whatever you tagged locally. In the second, intel/ is a Docker Hub namespace, so Docker goes to Docker Hub and ignores the image on your disk. An image you built yourself has no namespace prefix unless you gave it one.

docker run --rm --cap-add SYS_NICE inference-xeon-org-model:0.1.0 dry-run

If it is not listed at all, the name is wrong or you cannot reach the registry that has it.

Open your model's listing in the Intel® Software Catalog and copy the image name exactly: all lowercase, with a pinned tag. If you are serving, paste the listing's docker run. Do not derive the name from a Hugging Face id.

Then docker login — anonymous Docker Hub pulls are rate-limited, and a private registry on the listing needs those credentials. On Kubernetes, credentials mean a pull secret and imagePullSecrets on the pod spec (Deploy on Kubernetes). See also Prerequisites.

INFERENCE_MODEL_ID is required but not set

You started the base image without telling it which model to serve. Prefer a model image from the Intel® Software Catalog, which has the id baked in:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  "$IMAGE"

If you are using the base image, pass the Hugging Face id:

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_MODEL_ID="$MODEL_ID" \
  intel/inference-xeon-base:0.1.0

On Kubernetes the same variable goes under env: on the container.

No compatible profile found

Nothing built into this image matches the host, and this image will not serve untuned. Model images have general fallback switched off (INFERENCE_ALLOW_GENERAL_PROFILE_FALLBACK=false), so they stop here rather than starting on a configuration that was not tuned for the model. The base image, which has fallback on, would have started. See what was detected, then what would fit:

export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE "$IMAGE" detect-hardware
docker run --rm --cap-add SYS_NICE "$IMAGE" list-profiles

Three common causes:

• The host is not a supported Intel® processor (Intel® Xeon® CPU) (Supported Intel® platforms).
• A cgroup makes the container see the wrong core count. INFERENCE_ACCELERATOR_COUNT overrides it.
• Every row says numa_mismatch. Those configurations want more NUMA nodes than this host has.

To serve anyway, without the model-specific tuning, set INFERENCE_ALLOW_GENERAL_PROFILE_FALLBACK=true and accept unoptimized performance — or run the base image with INFERENCE_MODEL_ID instead (Environment variables).

On Kubernetes, a node selector keeps the pod on nodes whose capabilities match.

HuggingFaceHub API error: 401

Missing or invalid Hugging Face token, or the model's license is not accepted yet.

Gated models — Llama and Gemma — need both a Read token and you clicking Agree and access repository on the model page (gated models). A gated listing's docker run already includes -e HF_TOKEN. A 401 with a valid token almost always means the license is not accepted.

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e HF_TOKEN=<your-token> \
  intel/inference-xeon-meta-llama-llama-3.2-3b-instruct:0.1.0

Same command, with the cache mount, is in Quickstart. Swap the last line for your gated image name from the catalog. Keep the token out of shell history and out of the manifest — see Security.

Container OOM (exit code 137)

Not enough RAM for the model. 32GB is the floor for every model, whatever its size, and 8B and up want more; long context plus high concurrency need more again because the KV cache grows with both. See Supported Intel® platforms.

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 --shm-size=2g -m 64g \
  --cap-add SYS_NICE \
  "$IMAGE"

Three ways out, cheapest first: shrink the context with -e INFERENCE_ENGINE_ARGS='{"max-model-len": 4096}', give the container more memory, or pick a smaller model. On Kubernetes, set memory requests and limits so the pod lands on a node that can hold the model.

Available memory on node 0 ... less than desired CPU memory utilization

Different from the OOM kill above: the container exits with code 1 during engine startup — before it serves — and the machine may have plenty of free RAM overall. On the CPU backend vLLM reserves a fraction of a single NUMA node's total memory for weights and the KV cache, controlled by gpu-memory-utilization (a GPU carry-over name; on CPU it means CPU memory). The default is high (0.92), and startup requires that much to be free on that node — so any other load on the node makes it fail, even when other nodes or the machine as a whole have memory to spare.

Lower the fraction with INFERENCE_ENGINE_ARGS:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  -e INFERENCE_ENGINE_ARGS='{"gpu-memory-utilization": 0.5}' \
  "$IMAGE"

Confirm with dry-run --format json (look for --gpu-memory-utilization in "command_script"), then lower further if it still does not fit. See Environment variables.

FileNotFoundError: model not found in cache

Mount a writable cache directory, or let the engine download to the default path:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v /data/cache:/workspace/model-cache \
  "$IMAGE"

If you mount somewhere other than /workspace/model-cache, set INFERENCE_CACHE_PATH to the same path — a mismatch between the two is the usual cause. Pre-download: Model caching.

PermissionError: [Errno 13] Permission denied under hub/.locks/

The container reached the download step and could not write the cache. The traceback comes from the engine's Hugging Face client and names a .lock file, not the mount:

PermissionError: [Errno 13] Permission denied:
  '/workspace/model-cache/huggingface/hub/.locks/models--Qwen--Qwen3-4B/417d038a....lock'

The mounted directory is not writable by the container's non-root user — see Model caching for why and for other ways to grant access. Before the crash, the runtime logs:

WARNING: Cache directory is not writable by uid 1001:0 (...), but 'Qwen/Qwen3-4B'
is not cached at /workspace/model-cache or ... yet, so it has to be downloaded.

Fix the host directory, then start again:

chmod -R g+rwX /data/cache
find /data/cache -type d -exec chmod g+s {} +

On Kubernetes this is spec.template.spec.securityContext.fsGroup: 0 on the Pod. A cache that is complete can be mounted read-only — the engine only needs write access while weights are still being downloaded.

PermissionError: [Errno 13] Permission denied: '/.cache'

The weights loaded and then the engine died writing its compilation cache. The image sets HOME for its own uid, but an explicitly overridden HOME — for example -e HOME=... on docker run, or a Kubernetes env entry — takes precedence and can point somewhere the container's uid cannot write, so every cache the engine derives from HOME fails.

File ".../vllm/compilation/decorators.py", line 553, in __call__
  os.makedirs(inductor_cache, exist_ok=True)
PermissionError: [Errno 13] Permission denied: '/.cache'

Fixes, in order of preference:

1. Stop overriding HOME, or point it at a directory the uid can write (-e HOME=/tmp).
2. Keep the gid at 0 (--user <uid>:0, or runAsGroup: 0), which gives the uid write access to the image's own /home/model-server.

With any other gid the runtime relocates the caches under /tmp on its own and logs HOME (...) is not writable by uid .... That is a warning, not an error — serving continues, and only the compilation cache is lost between restarts.

curl: (7) Failed to connect to localhost port 8000: Connection refused

Nothing is listening where you are calling. In order of likelihood:

• The container exited. Check docker ps -a and the logs.
• You published no port. docker run needs -p 8000:8000.
• You set INFERENCE_PORT but published a different port. The env var, the -p mapping, and the URL must all use the same number — -p 9000:8000 without -e INFERENCE_PORT=9000 leaves the process on 8000 inside the container. See Environment variables.
• On Kubernetes, INFERENCE_PORT, containerPort, the Service port and targetPort, and both probes must all match.
• The image's EXPOSE is always 8000, so docker run -P publishes 8000 regardless of INFERENCE_PORT. Use an explicit -p <port>:<port>.

The image ships a Docker health check, so docker ps tells you whether the container thinks it is healthy before you debug the client side.

The model ... does not exist

The "model" field in your request does not match the model this container serves. Save the id the server reports:

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
echo "$MODEL_ID"

Use that exact string — it is the Hugging Face id, so it is case-sensitive and keeps its slash (Org/Model-Name), unlike the lowercase-and-hyphens image name. See API.

/health never returns 200, or I need more logs

Turn the framework's own logging up and leave the engine's where it is, so startup and configuration selection are visible without vLLM's output drowning them:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_LOG_LEVEL=DEBUG \
  -e INFERENCE_LOG_LEVEL_ROOT=INFO \
  "$IMAGE"

DEBUG prints what hardware was detected, which built-in configuration was selected, and the engine command that was generated. Levels: Logging. On Kubernetes: kubectl logs deployment/<name>, and kubectl describe pod <name> for probe failures and OOM kills.

A /health that stays 503 for several minutes on a first start is usually normal — weights are downloading. If it never turns 200 and the logs show no error, warm the cache first (Model caching) so you can tell a slow download apart from a real failure.

Check the configuration without loading weights

dry-run performs the same detection and selection as serve and prints the resolved configuration and the vLLM command, without downloading weights or binding the port. It is safe to run while another container is already serving.

export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE "$IMAGE" dry-run --format json

Look at "profile_id", "detection", and "command_script" to confirm an environment variable actually took effect. Commands: CLI.

numa_set_membind failed. errno: 1 / get_mempolicy: Operation not permitted

On a multi-socket host the engine binds each worker's memory to the CPU socket its threads run on, which keeps memory traffic local. The calls need CAP_SYS_NICE. Without it the startup log says:

NUMA memory binding is not permitted here (get_mempolicy: Operation not permitted), and this host has N NUMA nodes.

And the engine's own log shows:

get_mempolicy: Operation not permitted
[W utils.cpp:65] Warning: numa_set_membind failed. errno: 1

The container still serves, but workers' memory can sit on the wrong socket and pay cross-socket bandwidth. Grant the capability:

Docker:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  "$IMAGE"

Kubernetes: add it to the container securityContext.capabilities:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
    add: ["SYS_NICE"]

Full Deployment example: Deploy on Kubernetes — add the capabilities.add list above to its container securityContext. The grant works even though the container runs non-root: CAP_SYS_NICE only lifts the seccomp filter on set_mempolicy/get_mempolicy, and it grants no filesystem, network, or device privilege — see Security.

You may also see numa_migrate_pages failed. errno: 1 even with the capability granted — that one is harmless. The default seccomp profile blocks the migrate_pages syscall, which only relocates pages that were already allocated before the binding policy took effect. Subsequent allocations respect the policy, so the performance impact is negligible.

Slow first request, or not the throughput you expected

• The first request is slow. Weights are still downloading, or the engine is warming up. Warm the cache ahead of time (Model caching).
• Every request is slower than expected. Run list-profiles. If the [primary] row is a general rather than an optimized configuration, this host fell back and is not running the tuned settings for your model. numa_mismatch on the tp2 and tp4 rows means those need more sockets than you have. On multi-socket hosts, check for numa_set_membind failed in the log — see above.
• Throughput drops under load. Watch vllm:num_requests_waiting in metrics. A queue that keeps growing means you need more replicas, not different flags.
• Wrong hardware. detect-hardware shows what the container actually sees, including the cpuset it was given, which is often smaller than the machine.

Related pages

• Prerequisites — Docker login and Hugging Face token
• CLI — dry-run, list-profiles, and detect-hardware in detail
• Environment variables — every setting and how to verify it
• Logging — the two log-level knobs
• FAQ — short answers to common questions
• Supported features — before assuming something is broken, check it exists