Intel® Inference Microservices is a family of container images that serve a large language model tuned on Intel® processors, using the same API you already use with OpenAI. optimized by default.

You pick the model. At startup the container detects the Intel® processor (Intel® Xeon® CPU), applies the serving settings Intel® tuned and validated for that model and that silicon, and starts an OpenAI-compatible API. You do not hand-tune any parameter to get a working, optimized endpoint.

Choose how to run it

You can run it with Docker or on Kubernetes. It is the same container, and your app calls the same API.

Use the same image on Kubernetes when you need replicas.

Prerequisites

Do this once, before the first docker run. Full steps: Prerequisites.

At a glance

The API paths are /v1/chat/completions, /v1/completions, /v1/models, /health, and /metrics.

Start here

This is to serve Qwen3-4B. Paste it as-is — no Hugging Face token.

1. Start the server — leave this terminal open

The first start downloads the model and can take several minutes. Leave this terminal open — it is the server. Open a new terminal for step 2. Continue when you see that the model is alive.

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0

• --cap-add SYS_NICE — lets the engine pin memory to the CPU socket it is using.
• --shm-size=2g — Docker gives a container only 64 MiB of shared memory by default. The engine needs more than that to start, so without this flag the container exits. 2g is enough.
• -v is model caching: weights stay on disk so the next start skips the download. The path after the colon must be /workspace/model-cache/huggingface because the container is not root (Security).

2. New terminal — wait until it is alive, then chat

The first start can take several minutes while weights download. Run health until you see It's alive, then chat. This image is Qwen/Qwen3-4B — you do not copy an id from /v1/models.

curl -sf http://localhost:8000/health && echo "It's alive — the model is listening. Say hello."

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-4B",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Tip: Want to serve other models? Open the Intel® Software Catalog, find your model, and copy the docker run from the Deployment tab. If your model is not in the catalog, use the base image — Deploy an image that isn't in the catalog.

Documentation

Overview

• Meet Intel® Inference Microservices — what it is, why it exists, and model image versus base image

Getting started

• Prerequisites — hardware, Docker install and login, Hugging Face token and model approval
• Quickstart — serve a model with one docker run, wait on /health, send a chat request

Models and hardware

• Model catalog — validated model Docker images
• Supported Intel® platforms — which Intel® Xeon® CPU families and generations it runs on, and how much memory and disk each model size needs

Deployment

• Deploy on Kubernetes — Deployment and Service, probes, replicas, node selectors
• Model caching — download weights once so later starts skip the wait
• Security — keep port 8000 off the public network, store Hugging Face tokens as secrets

Configuration

• Environment variables — every INFERENCE_* setting
• Logging — turn the runtime up to DEBUG while keeping vLLM quiet

Reference

• API reference — the endpoints your application calls, streaming, framework integrations, Prometheus metrics
• CLI reference — serve, dry-run, list-profiles, download-to-cache, detect-hardware
• Supported features — what Intel® validated versus what passes through to vLLM
• Glossary — profile, NUMA, AMX, bf16, and the rest of the terms

FAQ and troubleshooting

• Troubleshooting — errors by message
• FAQ — short answers to the questions people ask first