Model Caching

← Docs index

Intel® Inference Microservices downloads model weights from Hugging Face the first time it serves a model. Mount a cache directory and the weights survive restarts, so the first request after a restart does not wait on that download — which for a large model is the difference between seconds and many minutes.

Two steps either way: run download-to-cache once to fill the directory, then mount that same directory when you serve.

The default cache path inside the container is /workspace/model-cache. Mount somewhere else and you must set INFERENCE_CACHE_PATH to the same path.

Each command that needs an image exports IMAGE — the last argument of the catalog listing's docker run (keep the tag). You only need "$MODEL_ID" on the base image — Environment variables. A named walkthrough is in Quickstart.

The mounted directory must be writable by the container user

The container runs non-root (uid 1001, gid 0 by default) and has to write the cache to download weights into it. A bind mount keeps the host directory's ownership, which overrides the ownership the image sets on /workspace/model-cache, so a directory that only your host account can write is a directory the container cannot write — this is true whether or not you are root on the host.

Prepare the directory once, before the first run:

bash

mkdir -p /data/cache && chmod 1777 /data/cache

1777 lets the container write regardless of who owns the directory, while the sticky bit stops other local users from deleting each other's cached files. Two alternatives, if a world-writable directory is not acceptable:

Option	Command	Notes
Grant the container's group	`chmod -R g+rwX /data/cache` then `find /data/cache -type d -exec chmod g+s {} +`	The container's gid is `0`, so this works on a `root`-owned directory. The `g+s` keeps new subdirectories in gid 0.
Match the host owner	`docker run --user "$(id -u):0" …`	Use when the directory is already owned by your host account. Any uid works. Keep the gid at `0` where you can: the image's own directories are group-writable through gid 0.

Getting this wrong shows up as PermissionError: [Errno 13] from deep inside the engine's Hugging Face client, on a .lock file under hub/.locks/. The runtime logs a warning naming the directory and the uid before launch — see Troubleshooting.

Warm the cache with `docker run`

Run download-to-cache against the directory you will later mount for serving:

        bash
        
        export IMAGE=<intel/inference-...:tag>

docker run --rm \
  --cap-add SYS_NICE \
  -v /data/cache:/workspace/model-cache \
  "$IMAGE" \
  download-to-cache

Then serve from the same mount:

        bash
        
        export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v /data/cache:/workspace/model-cache \
  "$IMAGE"

Gated models need -e HF_TOKEN=... on the download step too — the download is where the 401 happens. Skip HF_TOKEN when the listing's docker run has none. Only Hugging Face sources are supported (org/model or hf://org/model). --use-hf-cache writes Hugging Face's own hub/ layout instead of {cache_path}/{org}/{model}/, which is what you want if you are sharing a cache with other Hugging Face tooling. --model-id is only needed on the base image; a model image already has the id. Flags: CLI.

Reusing a cache you already have

Inside the container HF_HOME is /workspace/model-cache/huggingface, so an existing Hugging Face cache has to be mounted there — not at ~/.cache/huggingface, which nothing in the container reads:

        bash
        
        export IMAGE=<intel/inference-...:tag>

chmod -R g+rwX ~/.cache/huggingface
find ~/.cache/huggingface -type d -exec chmod g+s {} +

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  "$IMAGE"

It reuses whatever your local Hugging Face tooling already downloaded. Mount to that path, not /root/.cache/huggingface — the container is not root (Security). That mount also carries your huggingface-cli login token into the container (Security).

Cache weights on Kubernetes

Same two steps, split across two containers: an init container runs download-to-cache and exits, then the serving container starts with the weights already on disk, so the readiness probe passes much sooner. Full Deployment: Deploy on Kubernetes.

        yaml
        
      

        securityContext:
  fsGroup: 0            # on the Pod, not the container — makes a fresh PVC group-writable
initContainers:
- name: model-downloader
  image:                  # image name from the catalog
  args: ["download-to-cache"]
  env:
  # omit HF_TOKEN if the listing's docker run has none
  # - name: HF_TOKEN
  #   valueFrom:
  #     secretKeyRef:
  #       name: hf-credentials
  #       key: token
  volumeMounts:
  - name: model-cache
    mountPath: /workspace/model-cache
      

Use a PersistentVolumeClaim so the cache outlives the pod. With an emptyDir you pay the download on every reschedule.

A ReadWriteMany volume lets several replicas share one copy of the weights. Give write access to the init container only, and mount it read-only in the serving container, so no pod can rewrite weights another pod will load (Security).

How the cache path is resolved at serve time

{cache_path}/{org}/{model}/ is used if that directory exists.
Otherwise the bare model id is passed to vLLM, which resolves it through HF_HOME and the Hub as usual.

A missing local directory is not an error — it just means a download. That is why a typo in INFERENCE_CACHE_PATH shows up as an unexpectedly slow start rather than a failure. To confirm the path the runtime will use, run dry-run with INFERENCE_LOG_LEVEL=DEBUG and look for the cache path in the log lines.

CLI — download-to-cache — flags in full
Environment variables — INFERENCE_CACHE_PATH
Deploy on Kubernetes — where the init container goes
Security — read-only mounts once the cache is warm
Troubleshooting — model not found in cache, slow first request

PreviousDeploy on Kubernetes NextSecurity: network, tokens, and cache mounts