← Docs index

Which Intel® accelerators are supported today, and what a host needs to run Intel® Inference Microservices.

Supported families

This release is Intel® Xeon® CPU only.

You do not have to tune those instruction sets. The container detects them at startup and applies the matching configuration. What AMX, AVX-512, AVX2, and VNNI mean: Glossary.

Supported generations

The container does not pick settings from the product name. It looks at what the CPU can actually do: generation, cores, memory layout (NUMA), and instruction sets (AMX, AVX-512, AVX2, VNNI). Two machines with similar names can still get different settings if those capabilities differ.

An Intel® processor (Intel® Xeon® CPU) the runtime did not map to one of those rows is reported as xeon_generic. What each field means: Glossary.

What detection looks at

At startup the container inspects:

• Processor generation
• How many cores this container can use
• How many NUMA nodes the machine has
• Whether AMX / AVX-512 / AVX2 / VNNI are present

That combination selects a built-in configuration.

What happens when nothing in the image matches the host depends on which image you ran:

• A model image stops with No compatible profile found. It will not serve your model with untested settings.
• The base image falls back to a general configuration and serves, but without model-specific tuning.

Either way, start by seeing what was detected. detect-hardware is a command inside the container, not a host binary. It does not load a model, start the server, bind a port, or need a Hugging Face token. Use the base image:

docker run --rm --cap-add SYS_NICE intel/inference-xeon-base:0.1.0 detect-hardware

Family:       xeon
Model:        xeon_gnr
Cores:        16
Node Capacity: 192
NUMA Nodes:   2
Cpuset Bind:  0-15,64-79
Capabilities: amx=True, avx512=True, avx2=True, vnni=True

Family: xeon means Intel® Xeon® CPU. Model is an internal identifier for the generation the container found. Cores is the set of CPUs this container was given; Node Capacity is the whole host. A container restricted to 16 of 192 cores is configured for 16.

More output formats, including --format label for labelling Kubernetes nodes: CLI. What each field means: Glossary.

Requirements

Suggested minimum to try the product (4B–8B models, including the Quickstart). Larger models need more memory — see Memory and storage sizing.

Nothing else. No Python environment, no vLLM install, no compiler — the engine and its dependencies are inside the image.

The full platform matrix is in Supported features. First-run setup (Docker group, docker login, Hugging Face approval): Prerequisites.

Memory and storage sizing

Sizing follows the model, not the container.

At bf16, the weights alone need roughly 2GB per billion parameters. The KV cache grows on top of that with context length and concurrency — that is usually what causes an out-of-memory kill.

A mixture-of-experts model needs memory for all of its parameters even though only some are active per token. A 30B-A3B model is sized as a 30B model, not a 3B one.

Starting points, weights plus headroom for a modest context and low concurrency:

| Model size | Example| |---|---|---| | 2B–9B | 32GB | Qwen3-4B, Mistral-7B, Qwen3-8B, granite-3.2-2b | | 10B–16B | 48GB | Phi-4-reasoning | | 17B–24B | 64GB | gpt-oss-20b | | 25B–30B (including mixture-of-experts) | 96GB | Qwen3-30B-A3B |

Long context or many concurrent requests raises all of these. Measure with your own prompt lengths before you fix a limit.

• Storage: 50GB or more for the model cache, more if you serve several models from a shared volume. See Model caching.
• CPU: no hard minimum, and no upper one either — throughput scales with cores. On Kubernetes set CPU requests but not limits, and set memory requests equal to limits.

Lowering max-model-len through INFERENCE_ENGINE_ARGS is the cheapest way to cut memory when a model nearly fits — see Environment variables.

Keeping pods on matching nodes

Without a node selector, a pod can land on a node that matches no model-specific configuration.

• A model image fails at startup there and the pod restarts in a loop.
• The base image is quieter: it serves without model-specific tuning and reports no error.

Label your Intel® nodes (detect-hardware --format label produces the values) and pin the Deployment to them. See the production checklist in Deploy on Kubernetes.

Related pages

• Model catalog — validated model Docker images
• Deploy on Kubernetes — resource requests, node selectors, probes
• CLI — detect-hardware and list-profiles
• Troubleshooting — No compatible profile found, OOM