← Docs index

How the Intel® Inference Microservices container runs, and how to pass tokens and mount caches without leaking them. The container runs non-root, has no shell, and takes configuration only from environment variables. The endpoint it serves has no TLS and no authentication, so your access control lives in the network in front of it.

How the container runs

• Non-root by default.
• CAP_SYS_NICE only — --cap-add SYS_NICE on Docker, capabilities.add: ["SYS_NICE"] on Kubernetes. Lets the engine pin memory to the local CPU socket. See Troubleshooting.
• No shell — the engine is started by replacing the runtime process, not through sh -c, so there is no shell to inject into.
• Environment-only configuration — no extra CLI flags to smuggle values through. See Environment variables.
• No model-provided code, unless the model needs it — see below.

What this means for cache mounts: the container's Hugging Face cache lives at /workspace/model-cache/huggingface. If you are mounting a host directory to reuse an existing cache or huggingface-cli login token, mount it to that path — see Quickstart for the exact command.

Do not expose port 8000

The endpoint does not terminate TLS and does not authenticate callers, so anyone who can reach port 8000 can send requests, read the served model id, and scrape /metrics. Treat it as an internal service.

• Put an ingress or API gateway in front for TLS and authentication.
• In Kubernetes, add a NetworkPolicy so only the pods that should call it can.
• Do not give the Service type LoadBalancer or NodePort on a public network.

Passing a Hugging Face token safely

Only gated models need one — gated means Hugging Face will not download the weights until you accept the license. Llama and Gemma are gated; a gated listing's docker run already includes -e HF_TOKEN. Create the token and accept the model license first: Prerequisites. Never bake HF_TOKEN into an image, a committed file, or shell history. A cache mount can hand one over too — see Cache mounts.

# Bad — token in shell history
docker run -e HF_TOKEN=hf_abc123 ...

# Better — from a file you do not commit
docker run -e HF_TOKEN=$(cat .token) ...

# Best — a Kubernetes secret
env:
- name: HF_TOKEN
  valueFrom:
    secretKeyRef:
      name: hf-credentials
      key: token

Create the secret once:

kubectl create secret generic hf-credentials --from-file=token=.token

Wiring it into a Deployment: Deploy on Kubernetes. Rotating the token is an environment change plus a pod restart — you do not rebuild the image. The token is only needed while weights are being downloaded, so an init container that warms the cache can hold it while the serving container never sees it at all.

Models that execute their own code

A few models are not implemented by the engine itself. Loading one runs Python published in that model's Hugging Face repository — Hugging Face calls this remote code, and vLLM gates it behind --trust-remote-code. The code is imported while the model loads, which is before /health starts answering, and it runs as the container's main process.

The container grants this only where the model cannot load without it. Catalog model images carry the setting exactly where it is needed, and the startup log names the model that got it.

For a model you name yourself with INFERENCE_MODEL_ID on the base image, the container checks the model before starting the engine: it reads the model's own configuration from your cache if it is already there, otherwise it asks the Hugging Face Hub for the repository's file names and tags — no weights, no config download. If the model declares its own Python, the container enables execution and says so in the log, naming the evidence. If it does not, nothing is granted. The check never blocks startup: on a gated repository without a token, an offline host, or a network that does not answer, it logs that the answer was inconclusive and continues without the grant, and the engine's own error tells you if the model needed it.

To take the decision away from the image entirely:

export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -e INFERENCE_TRUST_REMOTE_CODE=false \
  "$IMAGE"

That is the hardened setting, and it wins over everything else — a model image's own configuration and INFERENCE_ENGINE_ARGS included. In Kubernetes, set it in the Deployment env: list so an admission policy can require it.

Cache mounts

Read-only, once the cache is warm, so a serving container cannot rewrite weights that other pods will load. On Kubernetes, give write access only to the init container that downloads. See Model caching.

Mounting your own Hugging Face directory — -v ~/.cache/huggingface:/workspace/model-cache/huggingface, as the Quickstart does — also mounts the token file that huggingface-cli login wrote. The container then authenticates as you even though HF_TOKEN was never set, which is why a gated model can work on your machine and fail everywhere else. On a development host that is fine. Anywhere shared, mount a dedicated cache directory and pass the token as a secret instead.

Related pages

• Prerequisites — Docker, login, Hugging Face token and model approval
• Deploy on Kubernetes — secrets, probes, and the production checklist
• Environment variables — every setting the container reads
• Model caching — the init-container pattern that isolates the token
• API — what an unauthenticated caller can reach