← Docs index

Intel® Inference Microservices exposes an OpenAI-compatible HTTP API on port 8000 once the model is loaded. Use GET /health to wait until it is ready, GET /v1/models for the served model id, POST /v1/chat/completions for chat, and GET /metrics for Prometheus.

The examples below use http://localhost:8000, which works against a Quickstart container on the same host, or a kubectl port-forward to the in-cluster Service. In a cluster, the base URL is http://<service-name>:8000/v1 — see Deploy on Kubernetes. The API is the same in both cases.

Snippets save the served id from GET /v1/models (no Python, no jq) — that copy is explained under INFERENCE_MODEL_ID. Each docker run that needs an image exports IMAGE (the last argument of the catalog listing's docker run). A named Qwen walkthrough is in Quickstart.

Endpoints

There is no authentication and no TLS on this port — see Security. Full request and response schemas are vLLM's: vLLM OpenAI-compatible server.

Health and readiness

GET /health returns an empty 200 OK once the engine has loaded the model, and 503 Service Unavailable while it is still starting or after it has failed. The body is empty (content-length: 0), so check the status code, not the response body.

curl -sf http://localhost:8000/health && echo ready

The first start can take several minutes while weights download and the engine warms up. Model caching removes the download from that time.

In Kubernetes, both probes hit the same path. Readiness gates traffic; liveness restarts a pod that went unhealthy after startup:

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 30

The image also carries a Docker HEALTHCHECK against the same path, so docker ps shows (health: starting) and then (healthy) when you run it with Docker.

The model id to pass

The chat API needs a "model" field. You do not copy that id from the catalog. Ask the server (GET /v1/models), save that "id" as MODEL_ID, then use it — Environment variables. One container serves one model, so there is a single "id". grep / cut are already on Linux — nothing to install.

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
echo "$MODEL_ID"

On the Quickstart image that prints Qwen/Qwen3-4B. Use "$MODEL_ID" as "model" in every request below.

Chat completions

POST /v1/chat/completions with the OpenAI request shape:

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "'"$MODEL_ID"'",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 128,
  "temperature": 0.7
}'

The response is the OpenAI shape too — the generated text is at choices[0].message.content:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "<MODEL_ID>",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello! How can I help you today?"},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 10, "completion_tokens": 9, "total_tokens": 19}
}

POST /v1/completions takes "prompt" instead of "messages" and returns the text at choices[0].text. Prefer chat completions for instruction-tuned models, which is most of the published catalog — they apply the model's chat template for you.

Streaming responses

Add "stream": true. Tokens arrive as they are generated instead of one JSON blob at the end.

Print the words (the chatbot view). Same Hello curl, piped so only content is printed as it arrives:

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
curl -N -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "'"$MODEL_ID"'",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": true
}' | awk -F '"content":"' 'NF>1 { split($2, a, "\""); printf "%s", a[1]; fflush() } END { print "" }'

The OpenAI SDK does the same thing:

import json
import urllib.request
from openai import OpenAI

model = json.load(urllib.request.urlopen("http://localhost:8000/v1/models"))["data"][0]["id"]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
stream = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
print()

The wire format is server-sent events: one data: JSON line per token, ending with data: [DONE]. curl without the awk prints that as-is — a full JSON object per token, not a chatbot. Drop the pipe when you need to inspect the protocol.

A couple of events look like this (fields omitted):

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{"content":"!"}}]}

data: [DONE]

A reasoning model (including Qwen3-4B) streams its deliberation in delta.content too, so a short prompt can produce thinking before the answer. A reasoning-parser moves that into reasoning_content — vLLM pass-through, see Supported features.

Streaming is the right default for interactive UIs: time to first token is much shorter than time to a full response, and vllm:time_to_first_token_seconds in metrics is what you watch for it.

Framework integrations

Any client that speaks the OpenAI API works once you point its base URL at the container. There is no custom protocol and no Intel® SDK to install. Two rules: model must be the id from GET /v1/models, and api_key is unused by the server but required by most clients, so pass any non-empty placeholder.

OpenAI Python SDK — the pattern every other client follows:

import json
import urllib.request
from openai import OpenAI

model = json.load(urllib.request.urlopen("http://localhost:8000/v1/models"))["data"][0]["id"]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Use each project's OpenAI or OpenAI-compatible provider docs from there. In a cluster, replace localhost:8000 with the Service name and port.

Tool calling

Tool calling is a vLLM feature and it is not enabled in the built-in configurations. To turn it on, pass the vLLM flags through INFERENCE_ENGINE_ARGS:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_ENGINE_ARGS='{"enable-auto-tool-choice": true, "tool-call-parser": "hermes"}' \
  "$IMAGE"

The parser must match the model's chat template — hermes above is an example, not a recommendation for every model. Check which parser vLLM provides for your model in the vLLM tool calling reference, then confirm the flags landed:

export IMAGE=<intel/inference-...:tag>

docker run --rm \
  --cap-add SYS_NICE \
  -e INFERENCE_ENGINE_ARGS='{"enable-auto-tool-choice": true, "tool-call-parser": "hermes"}' \
  "$IMAGE" \
  dry-run --format json

Once enabled, requests use the standard OpenAI tools and tool_choice fields and the response carries choices[0].message.tool_calls. Anything you set this way is outside the Intel®-validated configuration for that model — see Supported features.

Structured output

Also a vLLM feature. Select a structured-outputs backend with INFERENCE_ENGINE_ARGS, then use the per-request response_format field. Save the served id first, then put it in "model":

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_ENGINE_ARGS='{"structured-outputs-config": {"backend": "xgrammar"}}' \
  "$IMAGE"

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "'"$MODEL_ID"'",
  "messages": [{"role": "user", "content": "Give me a user record."}],
  "response_format": {"type": "json_object"}
}'

Backend names and the supported subset of JSON Schema are vLLM's: see the vLLM structured outputs reference. As with tool calling, this is a pass-through, not a validated configuration.

Error responses

Error bodies follow the OpenAI error shape ({"error": {...}}), because the HTTP layer is vLLM's.

Prometheus metrics

GET /metrics on the same port as chat, in Prometheus text format, as soon as the engine is up. Metric names use a colon, not an underscore: vllm:num_requests_running, not vllm_num_requests_running.

curl -s http://localhost:8000/metrics

The signals worth alerting on:

curl -s http://localhost:8000/metrics | grep -E 'vllm:(num_requests_running|num_requests_waiting|prompt_tokens_total|generation_tokens_total|time_to_first_token_seconds|e2e_request_latency_seconds)'

vllm:num_requests_running{engine="0",model_name="<MODEL_ID>"} 0.0
vllm:num_requests_waiting{engine="0",model_name="<MODEL_ID>"} 0.0

Labels such as model_name carry the served id, so one dashboard can span several Deployments. Counters stay at 0 until you send a request. vLLM's own # HELP text may mention GPU; on Intel® Xeon® CPU these are still the running and waiting request counts.

To scrape it, point Prometheus at the Service from Deploy on Kubernetes:

scrape_configs:
  - job_name: inference
    metrics_path: /metrics
    static_configs:
      - targets: ['inference:8000']  # in-cluster Service name:port
      # - targets: ['localhost:8000']              # kubectl port-forward or local Docker

Related pages

• Quickstart — get an endpoint to call these against
• Deploy on Kubernetes — Service name as the base URL
• Supported features — validated versus pass-through capabilities
• Security — why not to expose port 8000
• Troubleshooting — errors by message