← Docs index

Run a model image as a Kubernetes Deployment and Service. You get replicas, restarts, readiness checks, secrets, and rolling updates.

It is the same image the Quickstart runs with docker run. The cluster adds operations around it, not a different container.

Apply a manifest, wait until /health is ready, then call /v1/chat/completions through the Service (or kubectl port-forward while you test).

Prerequisites: a Kubernetes cluster with Intel® worker nodes, kubectl access, and the image from the Intel® Software Catalog. For gated models, create a Hugging Face token and accept the license first (Prerequisites), then an hf-credentials secret

What to change for your model

The YAML is generic. Paste the image name from the listing's docker run in the Intel® Software Catalog (a named Docker example is in Quickstart):

Pin the image tag (:0.1.0). Do not use :latest. Other settings (INFERENCE_PORT, INFERENCE_ENGINE_ARGS, …) use the same names as Environment variables under env:.

Manifest

Save this as inference.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference                        # change me
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inference                     # must match template labels
  template:
    metadata:
      labels:
        app: inference                   # must match selector
    spec:
      securityContext:
        runAsNonRoot: true
        fsGroup: 0                       # makes a mounted volume group-writable for the container
      containers:
        - name: inference-runtime
          image:                  # image name from the catalog
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
              add: ["SYS_NICE"]   # same as docker --cap-add SYS_NICE; NUMA memory pinning, not extra privilege
          ports:
            - containerPort: 8000
          # env:  # uncomment for gated models (HF_TOKEN). INFERENCE_MODEL_ID is base image only
          #   - name: HF_TOKEN
          #     valueFrom:
          #       secretKeyRef:
          #         name: hf-credentials
          #         key: token
          #   - name: INFERENCE_MODEL_ID
          #     value: "<MODEL_ID>"      # Hugging Face id; base image only
          #   # On a cluster without direct egress, the weight download needs a proxy. A pod
          #   # inherits nothing, so set these on every container that reaches the internet -
          #   # init containers included.
          #   - name: http_proxy
          #     value: "http://proxy.example.com:911"
          #   - name: https_proxy
          #     value: "http://proxy.example.com:912"
          #   - name: no_proxy
          #     value: "localhost,127.0.0.1,.svc,.svc.cluster.local"
          resources:
            requests:
              cpu: "8"
              memory: 32Gi
          volumeMounts:
            - name: dshm                 # same as docker --shm-size=2g
              mountPath: /dev/shm
          # Loading a model takes minutes. startupProbe covers that window; readiness
          # and liveness only start counting once it has succeeded.
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 15
            failureThreshold: 80                
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 30
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: inference                        # change me
spec:
  selector:
    app: inference                       # must match pod labels
  ports:
    - port: 8000
      targetPort: 8000

Apply the manifest

kubectl apply -f inference.yaml
kubectl rollout status deployment/inference
kubectl get pods -l app=inference

Wait until the pod is Ready. The readiness probe hits /health; the first start can take several minutes while weights load.

If the image is private, create a pull secret and add imagePullSecrets on the pod spec. If the model is gated, create the Hugging Face secret first:

kubectl create secret generic hf-credentials --from-file=token=.token

Then uncomment HF_TOKEN in the manifest. Do not put the token in the YAML. See Security.

Call the API

From outside the cluster (test):

kubectl port-forward svc/inference 8000:8000

Leave that running. Open a new terminal. The first start can take several minutes. Run health until you see It's alive, then chat. "model" is the id this pod is serving — save it from GET /v1/models (Environment variables).

curl -sf http://localhost:8000/health && echo "It's alive — the model is listening. Say hello."

MODEL_ID=$(curl -s http://localhost:8000/v1/models | grep -o '"id": *"[^"]*"' | head -1 | cut -d '"' -f4)
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "'"$MODEL_ID"'",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

From another pod in the cluster: http://inference:8000/v1 (Service name + port). Put ingress or a gateway in front for TLS — do not expose port 8000 to the internet. See Security and API.

Setting environment variables

Same names as on Docker, under env: on the container. Example:

env:
  - name: INFERENCE_ENGINE_ARGS
    value: '{"max-model-len": 4096}'
  - name: INFERENCE_LOG_LEVEL
    value: "DEBUG"

Full list: Environment variables. Check a value with dry-run before you put it in the manifest.

Production checklist

The manifest above is the minimum that deploys. Before production:

• Resource requests: raise the 32Gi in the manifest to the figure for your model — see Memory and storage sizing — so the pod lands on a node that can serve it. 32Gi is a floor, not an estimate for a small model: below it the engine's startup reservation fails whatever the model size, and the pod exits with a ValueError before it serves (Troubleshooting).
• Proxy: on a cluster without direct egress, uncomment the proxy variables in the manifest and set them on every container that downloads weights, including init containers. A pod inherits nothing from the host.
• Node selectors / tolerations: Pin to Intel® nodes this image supports. Without a selector, the pod can land on a node whose capabilities match no built-in config. A model image fails at startup there and the pod restarts in a loop; the base image serves without model-specific tuning and reports nothing.
• Replicas: replicas: 1 is a starting point. Keep the readiness probe on /health so traffic only shifts to pods once the engine has loaded the model.
• Model cache: warm weights in an init container, then serve from the same volume — Model caching.
• Security context: the manifest above sets the minimum the container needs. runAsUser/runAsGroup can be set to any values — keeping runAsGroup: 0 lets the container reuse the writable directories already in the image; any other gid falls back to a scratch directory under /tmp with a warning logged. readOnlyRootFilesystem: true needs an emptyDir mounted at /tmp for that fallback to have somewhere to write — see Troubleshooting.
• Cache volume: keep fsGroup: 0 while a volume still has to be written; once the cache is warm, mount it readOnly instead — see Model caching.

Readiness, liveness and startup probes

All three hit /health, and each has one job:

Raise failureThreshold for large models, or warm the cache so loading no longer competes with a download.

Related pages

• Model catalog — validated model Docker images
• Environment variables — every name that can go under env:
• Model caching — init container plus a persistent volume
• Security — secrets, NetworkPolicy, why not to expose port 8000
• API — what to call through the Service, and metrics to scrape
• Supported Intel® platforms — resource requests and keeping pods on matching nodes
• Troubleshooting — pods stuck not-ready, OOM kills, image pull failures