software catalog

Updated Sep 1, 2026

Frequently Asked Questions

← Docs index

Short answers about Intel® Inference Microservices.

On this page

What is Intel® Inference Microservices?

A container that serves one large language model on Intel® processors, through an OpenAI-compatible HTTP API.

At startup it looks at the Intel® processor (Intel® Xeon® CPU) (generation, cores, memory layout, instruction sets), matches that to a built-in configuration for your model, and starts vLLM.

What problem does it solve?

Serving settings depend on the Intel® processor (Intel® Xeon® CPU) and the model. Intel® Inference Microservices ships tested values inside the image and applies them at startup, so the same docker run or Kubernetes Deployment is configured for that host.

Which image should I run: model or base?

Image kindExampleUse it whenYou pass
Model imagepaste the listing's docker run from the Intel® Software CatalogThe catalog lists your modelNothing extra
Base imageintel/inference-xeon-base:0.1.0Your model is not in the catalog, or you are not sure it isINFERENCE_MODEL_ID — Environment variables

Prefer the model image: it is the base image plus that model's validated configuration. See If your model is in the catalog.

Where do I find the image name?

Paste the docker run from the model's listing in the Intel® Software Catalog. That command already has the image and, for gated models , you will see -e HF_TOKEN included.

The image name (intel/inference-...:tag) is the last argument of that command. You need it alone for Kubernetes image: and for dry-run / list-profiles — see CLI and Environment variables.

Do not build the name from the Hugging Face id yourself. A guessed name fails with pull access denied. See Troubleshooting.

How do I run it with 1 command?

Paste the docker run from the model's listing in the Intel® Software Catalog, or the Qwen3-4B command in Quickstart. Wait for curl -sf http://localhost:8000/health to return 200, then call POST /v1/chat/completions. If the cache mount fails with a permission error: Model caching.

Should I run it with Docker or on Kubernetes?

docker run is the fastest way to get a working endpoint on one machine.

Kubernetes is what you want once you need replicas, restarts, readiness checks, secrets, and rolling updates.

It is the same container, and your app calls the same API. Moving from Docker to Kubernetes changes a manifest, not your application. See Quickstart and Deploy on Kubernetes.

Is there a Helm chart or an Operator?

Not yet. Until then, apply the Deployment and Service in Deploy on Kubernetes.

What API does it expose?

An OpenAI-compatible API:

  • POST /v1/chat/completions — chat, including streaming
  • POST /v1/completions — text completion
  • GET /v1/models — the id of the one model this container serves
  • GET /health — 200 when ready, 503 when not
  • GET /metrics — Prometheus metrics

Details: API reference.

Does it work with the OpenAI SDK, LangChain, or LlamaIndex?

Yes. Set the client's base URL to http://localhost:8000/v1 (local) or http://<service>:8000/v1 (in-cluster). There is no custom protocol.

api_key is unused by the server, but most clients require a non-empty string, so any placeholder works. Snippets: API reference.

Does it support streaming responses?

Yes. Set "stream": true on /v1/chat/completions. Print only the words as they arrive (see Streaming); without that, curl dumps a JSON object per token.

Does it support tool calling and structured output?

Both are vLLM features and are off by default. Turn them on with INFERENCE_ENGINE_ARGS. The result is yours, not the Intel®-validated configuration. See Tool calling and Structured output.

Can one container serve more than one model?

No. One container serves exactly one model. To serve several models, run one Deployment per model and route between them in your application or with a gateway such as LiteLLM.

Which Intel® processors (Intel® Xeon® CPU) are supported?

Intel® Xeon® 4 processor, Intel® Xeon® 5 processor, or Intel® Xeon® 6 processor (Sapphire Rapids, Emerald Rapids, Granite Rapids, Sierra Forest). To see what this machine is, run detect-hardware inside the container — command, sample output, and what Family: / Model: mean: Supported Intel® platforms.

Does it run on Intel® GPUs, NPUs, or non-Intel® CPUs?

Not today. The published images target Intel® Xeon® CPU with vLLM. See Supported features.

How much memory does it need?

32 GB is the suggested minimum for every model, whatever its size. Bigger models need more — 48 GB around 14B, 64 GB around 20B, 96 GB at 30B. Plan 50 GB or more of disk for the model cache. Longer context and higher concurrency need more RAM. See Supported Intel® platforms.

Do I need a Hugging Face token?

Only for gated models — Hugging Face models whose publisher blocks the download until you accept the license on the model page. Llama and Gemma are gated; Qwen, Mistral, Granite, Phi, and GPT-OSS are not. A gated listing's docker run already includes -e HF_TOKEN.

When it is Yes, you need two things: a Read token, and you must open the model page and accept its license. A token without that approval still returns 401. See Prerequisites and Security.

Why is the first request so slow?

The first start downloads model weights from Hugging Face and warms the engine. That can take several minutes.

Pre-download the weights with download-to-cache — on Kubernetes, in an init container — and later starts skip the download. See Model caching.

How do I check the configuration without loading the model?

Run dry-run. It does the same hardware detection and configuration selection as serve, then prints what it would run. It does not download weights and does not bind port 8000. IMAGE is the last argument of the catalog listing's docker run:

bash 
export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE "$IMAGE" dry-run --format json

See CLI reference.

What commands does the container accept?

Five: serve (the default), dry-run, list-profiles, download-to-cache, and detect-hardware. Everything else is INFERENCE_* environment variables. Names are single tokens — list-profiles, not list profiles. See CLI reference.

Can I override the engine flags Intel® picked?

Yes, with INFERENCE_ENGINE_ARGS, a JSON object merged over the built-in configuration — for example {"max-model-len": 4096}. Anything you override is no longer the validated configuration. Check with dry-run first. See Environment variables.

What happens if my host does not match any built-in configuration?

A model image stops with No compatible profile found. The base image falls back to a general configuration and serves without model-specific tuning.

See what the container found on the base image, then which configurations in your model image fit this host (neither command starts the server):

bash 
export IMAGE=<intel/inference-...:tag>

docker run --rm --cap-add SYS_NICE intel/inference-xeon-base:0.1.0 detect-hardware
docker run --rm --cap-add SYS_NICE "$IMAGE" list-profiles

IMAGE is the last argument of the catalog listing's docker run. Details: Supported Intel® platforms and CLI.

Is the endpoint safe to expose to the internet?

No. Port 8000 has no TLS and no authentication. Treat it as an internal service. Put an ingress or gateway in front. See Security.

How do I monitor it?

Scrape GET /metrics with Prometheus. Useful signals are queue depth (num_requests_waiting), time to first token, and end-to-end latency. See API reference.

Why do the docs say :0.1.0 instead of :latest?

A pinned tag keeps a rollout reproducible. :latest can change the engine and the built-in configuration between two pulls. Always pin the tag shown on the listing.

What license is it under, and what about the models?

The project is Apache-2.0. Each model carries its own license, on that model's listing in the Intel® Software Catalog. You are responsible for complying with the license of the model you serve.

Related pages

PreviousSupported FeaturesNextTroubleshooting