Model Caching

← Docs index

By default, Intel® Inference Microservices downloads model weights from Hugging Face when the container starts. Without a persistent cache, those weights must be downloaded again on subsequent container runs.

Model caching lets you download the model once and reuse the locally cached weights across runs. This can significantly reduce startup time, especially for larger models.

Caching requires two steps:

Run download-to-cache once to download the model into a persistent directory.
Mount that same directory when starting the inference container.

The default cache path inside the container is /workspace/model-cache. If you mount the cache at a different path, set INFERENCE_CACHE_PATH to that path.

For the commands below, set IMAGE to the container image and tag shown in the catalog. MODEL_ID is required only when using the base image. See Environment variables. For a complete example, see the Quickstart.

The mounted directory must be writable by the container user

The container runs as a non-root user (UID 1001, GID 0 by default) and must be able to write model files to the cache directory.

With a bind mount, the mounted directory keeps its host ownership and permissions. These permissions replace those configured for /workspace/model-cache inside the image. As a result, a host directory that is not writable by UID 1001 or GID 0 can prevent the container from downloading the model.

Prepare the directory before the first run:

bash

mkdir -p /data/cache && chmod 1777 /data/cache

1777 allows the container to write to the directory regardless of ownership, while the sticky bit prevents users from deleting files owned by other users.

If a world-writable directory is not appropriate for your environment, use one of the alternatives below:

Option	Command	Notes
Grant write access to the container group	`chmod -R g+rwX /data/cache` `find /data/cache -type d -exec chmod g+s {} +`	The container uses GID `0` by default. This works well when the cache directory is owned by the `root` group. The setgid bit (`g+s`) keeps new directories in the same group.
Run with the host UID	`docker run --user "$(id -u):0" …`	Useful when the cache directory is already owned by your host user. Keeping GID `0` allows access to directories in the image that are group-writable.

If the cache is not writable, the container logs a warning before startup and the model download may fail with a permission error, often on a lock file under hub/.locks/.

See Troubleshooting.

Warm the cache with Docker

Download the model (download-to-cache) once into the directory you will use as the persistent cache:

        bash
        
        export IMAGE=<intel/inference-...:tag>

docker run --rm \
  --cap-add SYS_NICE \
  -v /data/cache:/workspace/model-cache \
  "$IMAGE" \
  download-to-cache

Then mount the same directory when starting the inference service:

        bash
        
        export IMAGE=<intel/inference-...:tag>

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  --shm-size=2g \
  -v /data/cache:/workspace/model-cache \
  "$IMAGE"

The model is now reused from the local cache instead of being downloaded again on each container run.

Gated models: Add -e HF_TOKEN=... to the download-to-cache command when the model requires Hugging Face authentication. You do not need HF_TOKEN for public models. For the base image, specify the model with --model-id. Model-specific images already include the model ID. See the download-to-cache CLI reference for additional options.

Reusing a cache you already have

If the model is already present in your local Hugging Face cache, you can mount that cache directly instead of downloading it again. Inside the container, the Hugging Face cache is located at:

        bash
        
        export IMAGE=<intel/inference-...:tag>

chmod -R g+rwX ~/.cache/huggingface
find ~/.cache/huggingface -type d -exec chmod g+s {} +

docker run --rm -p 8000:8000 \
  --cap-add SYS_NICE \
  -v ~/.cache/huggingface:/workspace/model-cache/huggingface \
  "$IMAGE"

It reuses whatever your local Hugging Face tooling already downloaded. Mount to that path, not /root/.cache/huggingface — the container is not root (Security). That mount also carries your huggingface-cli login token into the container (Security).

Cache weights on Kubernetes

The same approach applies to Kubernetes:

An init container runs download-to-cache.
The serving container starts after the model is available on the shared volume.

Use a PersistentVolumeClaim so the model cache survives pod restarts and rescheduling.

        yaml
        
      

        securityContext:
  fsGroup: 0            # on the Pod, not the container — makes a fresh PVC group-writable
initContainers:
- name: model-downloader
  image:                  # image name from the catalog
  args: ["download-to-cache"]
  env:
  # omit HF_TOKEN if the listing's docker run has none
  # - name: HF_TOKEN
  #   valueFrom:
  #     secretKeyRef:
  #       name: hf-credentials
  #       key: token
  volumeMounts:
  - name: model-cache
    mountPath: /workspace/model-cache
      

For gated models, provide HF_TOKEN to the init container using a Kubernetes Secret. An emptyDir does not persist across pod rescheduling, so the model may need to be downloaded again. A persistent volume avoids that repeated download. See Deploy on Kubernetes for the complete deployment example.

How the cache path is resolved at serve time

At startup, the runtime checks for a locally cached model first:

If {cache_path}/{org}/{model}/ exists, the local model is used.
Otherwise, the model ID is passed to the inference engine, which resolves it through Hugging Face.

A missing cache is not an error. The model is downloaded instead. If a cached model is unexpectedly downloading again, verify INFERENCE_CACHE_PATH and run with INFERENCE_LOG_LEVEL=DEBUG to inspect the resolved cache path.

CLI — download-to-cache — flags in full
Environment variables — INFERENCE_CACHE_PATH
Deploy on Kubernetes — where the init container goes
Security — read-only mounts once the cache is warm
Troubleshooting — model not found in cache, slow first request