← Docs index

Terms used across the Intel® Inference Microservices documentation, and what each one means in this product specifically. Terms are grouped by what you are looking at: images, startup and configuration, hardware, and serving.

Images and the catalog

Intel® Inference Microservices — Intel®'s container for serving one large language model on Intel® processors through an OpenAI-compatible API. It detects the host at startup and applies a built-in serving configuration for the hardware and model pair.

Base image — intel/inference-xeon-base:<tag>. The shared runtime with general, non-model-specific configurations. It has no model baked in, so you must pass INFERENCE_MODEL_ID. Use it when the Intel® Software Catalog has no image for your model.

Model image — intel/inference-<accelerator>-<org>-<model>:<tag>. The base image plus that model's Intel®-validated configurations, with the model id already set. Nothing extra to pass. Paste the docker run from the Intel® Software Catalog.

Image name — always lowercase. Looks like intel/inference-xeon-org-model:0.1.0. It is the last argument of the catalog listing's docker run. Hugging Face ids keep Org/Model-Name; image names are lowercase with hyphens, for example org/Model-Name is served by intel/inference-xeon-org-model-name.

Intel® Software Catalog — swcatalog.intel.com, where every validated model is listed. Paste the listing's docker run to serve. The image name at the end of that command is what these docs put in IMAGE for dry-run, list-profiles, and Kubernetes image:.

Tag — the version, for example :0.1.0. Pin it. :latest can change the engine version and the built-in configuration between two pulls.

Model id — the Hugging Face repository id of the model being served (Org/Model-Name). It is what GET /v1/models returns and what you pass as "model" in a request. A named example is in Quickstart.

Gated model — a Hugging Face model whose publisher will not let you download the weights until you accept the model license on the model page. Serving one requires a Hugging Face Read token (HF_TOKEN). Llama and Gemma are gated; Qwen, Mistral, Granite, Phi, and GPT-OSS are not. A gated listing's docker run already includes -e HF_TOKEN. Hugging Face's own explanation: gated models.

Startup and configuration

Built-in configuration (profile) — a validated set of engine settings shipped inside the image for one hardware-and-model pair. The docs call it a built-in configuration; the CLI and environment variables call it a profile, which is why you see list-profiles and INFERENCE_PROFILE_ID. You do not author profiles — you only select or inspect them.

Profile id — the identifier of one built-in configuration, for example vllm-xeon-bf16-tp1. list-profiles prints them; INFERENCE_PROFILE_ID pins one instead of letting the runtime pick.

[primary] — the marker list-profiles puts next to the configuration the runtime would choose for this host. If you pin nothing, this is what serve uses.

Optimized profile — a configuration tuned for a specific model on a specific Intel® processor (Intel® Xeon® CPU) generation. This is what a model image adds over the base image.

General profile — a configuration that fits the hardware but is not tuned for your specific model. It keeps the container serving when no optimized configuration matches, but only where fallback is allowed — the base image, not a model image.

Profile fallback — when nothing model-specific matches the host, the runtime can drop to a general configuration instead of refusing to start. INFERENCE_ALLOW_GENERAL_PROFILE_FALLBACK controls it: true on the base image, false on model images.

numa_mismatch — the reason list-profiles gives for an incompatible row: that configuration wants more NUMA nodes (in practice, more sockets) than this host has. A tp4 row on a 2-socket host is a typical case.

Detection — the startup probe of the host: processor generation, core count, NUMA node count, cpuset binding, and AMX / AVX-512 / AVX2 / VNNI capability flags. Run it on its own with detect-hardware.

Dry run — dry-run: the same detection and configuration selection as serve, but it prints the resolved configuration and the vLLM command instead of starting the engine. No weights are downloaded and port 8000 is not bound.

Engine args — the vLLM command-line flags the built-in configuration produces. INFERENCE_ENGINE_ARGS merges a JSON object of your own over them, for example {"max-model-len": 4096}.

Model cache — a directory of downloaded model weights, /workspace/model-cache inside the container by default. The Hugging Face layout lives at /workspace/model-cache/huggingface because the container runs non-root — mount a host cache to that path. Warm it with download-to-cache so the first request does not wait on a Hugging Face download. See Model caching.

Hardware terms

Accelerator family — the class of silicon the runtime targets. Today the only published value is xeon (Intel® Xeon® CPU).

Accelerator model — the processor generation detection reports, and the value INFERENCE_ACCELERATOR_MODEL overrides. These are internal identifiers:

Accelerator count — the number of CPU cores the container can actually use (the cpuset it was given, not the whole socket). INFERENCE_ACCELERATOR_COUNT overrides it when a cgroup makes detection report the wrong number.

AMX (Intel® Advanced Matrix Extensions) — a matrix-multiply accelerator inside each CPU core. An LLM spends most of its time multiplying matrices, and AMX does that in dedicated hardware rather than in vector instructions, for bf16 and int8. On Intel® Xeon® 4 processor, Intel® Xeon® 5 processor, and Intel® Xeon® 6 processor with P-cores; an E-core host (xeon_srf) reports amx=False.

AVX-512(Intel® Advanced Vector Extensions)— 512-bit vector instructions that covers the work around the matrix multiplies — attention softmax, normalization, sampling, KV cache movement — so they matter even where AMX is present.

AVX2 and VNN—— vector instruction sets detection also reports. AVX2 is 256-bit vectors, half the width of AVX-512; VNNI adds int8 dot products. They matter to the configuration chosen for hosts without AMX, or for parts of the model that do not use AMX.

--cap-add SYS_NICE / CAP_SYS_NICE — lets the engine pin memory to the CPU socket it is using. Docker and Kubernetes must grant it. If it is missing: Troubleshooting.

NUMA node — a memory locality domain, usually one per socket. Crossing NUMA nodes for weights or KV cache costs bandwidth, which is why configurations bind cores and why some configurations need a minimum node count.

Node capacity — the total core count of the host, as opposed to the cpuset the container was given. detect-hardware prints both so you can see when a container is restricted.

Tensor parallelism (tp1, tp2, tp4) — how many shards the model is split across. On Intel® Xeon® CPU, shards map to NUMA nodes, so tp2 needs two nodes and tp4 needs four. This is where numa_mismatch comes from.

bf16 (bfloat16) — the 16-bit floating-point format these configurations use on Intel® processors (Intel® Xeon® CPU). INFERENCE_PRECISION accepts bf16 or auto, which resolves to the same thing today.

Serving terms

vLLM — the inference engine inside the container. It provides the OpenAI-compatible HTTP server and the Prometheus metrics. Intel® Inference Microservices decides how to launch it; vLLM handles the requests. Request and response shapes are documented in the vLLM OpenAI-compatible server reference.

OpenAI-compatible API — the request and response shape of the OpenAI Chat Completions API, served over plain HTTP. Any OpenAI client works by changing its base URL to the container. No custom protocol, no vendor SDK.

KV cache — the per-request attention state vLLM keeps while generating. It grows with context length and concurrency, and it is the reason memory needs exceed the size of the weights.

Context length (max-model-len) — the maximum prompt plus completion tokens a request may use. Lowering it with INFERENCE_ENGINE_ARGS reduces KV-cache memory.

Exec replace — how serve hands off: the runtime process is replaced by the engine process rather than supervising it. That is why the container has no shell wrapper and why configuration is read once, at startup.

Readiness vs liveness — both Kubernetes probes hit /health. Readiness decides whether the Service sends traffic, so it needs a delay long enough for weights to load. Liveness restarts a pod that became permanently unhealthy after startup. See Deploy on Kubernetes.

Related pages

• FAQ — short answers to the common questions
• CLI — the five commands these terms show up in
• Environment variables — every INFERENCE_* setting
• Supported Intel® platforms — Intel® processors (Intel® Xeon® CPU) and sizing
• Supported features — what works today