Intel® Inference Microservices

One container that matches your hardware to the model, applies the validated runtime profile, and serves it behind an OpenAI-compatible API.

Overview

Serving an LLM well on a given accelerator means getting dozens of flags right — batch size, precision, core affinity, KV-cache settings — and getting them wrong costs throughput or stability. Most teams tune this by hand, per model, per box.

Intel® Inference Microservices packages this as a Docker container that resolves it all at startup: it matches the running hardware to the model being served and picks the validated profile that performs best on that hardware/model pair, so the engine launches already correctly configured. The server it launches speaks the OpenAI-compatible API, so any existing OpenAI client or SDK works against it by changing the base URL.

Model catalogBrowse every model served by Intel Inference Microservices.Hardware compatibilitySupported processors and validated image matrix.Full documentationDeployment, customization, and reference guides.

Usage

The fastest way to see what this is: start a serving container, then hit its OpenAI-compatible API.

Validate prerequisites before running the commands.

Start the server

Choose one of the following options:

Use a validated catalog model

Recommended

If you know the Intel-provided image for your model (it starts with intel/), pull and run the validated and optimized image directly. Browse the validated model catalog.

bash

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  intel/inference-xeon-qwen-qwen3-4b:0.1.0

Call the endpoint

In another terminal, wait until the container reports itself ready:

bash

curl http://localhost:8000/health

Then send a request:

bash

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Any OpenAI-compatible SDK works the same way — set the base URL to http://localhost:8000/v1 For frameworks such as LangChain, see the OpenAI-compatible API guide.