What is it?

models.yaml is the one file you edit to control what runs and how. It is the single source of truth behind every model-manager command — you rarely touch anything else.

The installer seeds env/<env>/models.yaml from this repo's model_manager/models.yaml. Edit the environment copy to add or remove models; the repo copy is the template for new environments.

It has three parts:

defaults:            # 1. Fallbacks applied to every model (namespace, engine, replicas)
storage:             #    Where downloaded weights live
network:             #    Proxy / connectivity settings for pulling from Hugging Face

models:              # 2. The models you can deploy — one entry per model
- name: qwen3-0-6b
  model_id: Qwen/Qwen3-0.6B     # the Hugging Face repo to pull
  category: llm                 # workload type → sensible CPU/memory/flags
  servers:                      # which engine + versions this model may use
    vllm: { versions: ["0.24.0", "0.19.1"], default: "0.24.0" }

runtimes:            # 3. The engines themselves — images, versions, shared settings
  vllm: { ... }
  openvino: { ... }

What that gives you:

Global sections

defaults

Applied to every model that doesn't override them.

storage

network

Used by the download Job when pulling from Hugging Face. Empty values inherit from the installer's environment.

Per-model fields

Only name and model_id are required.

models:
- name: qwen3-0-6b
  model_id: Qwen/Qwen3-0.6B
  category: llm
  cpu: 8
  memory: 16Gi

Plus the server binding, described next.

Choose which server and version a model may use

Two equivalent forms, depending on how much control you want:

# Preferred: declare the versions this model is known to work with (an allow-list)
- name: qwen3-0-6b
  model_id: Qwen/Qwen3-0.6B
  category: llm
  servers:
    vllm: { versions: ["0.24.0", "0.19.1"], default: "0.24.0" }
  default_server: vllm

# Shorthand: one server, optionally pinned to a version — no allow-list
- name: llama3-8b-awq
  model_id: casperhansen/llama-3-8b-instruct-awq
  category: llm
  runtime: vllm
  server_version: "0.24.0"

With the servers: form, a deploy that asks for a version not in the list is rejected, so nobody accidentally ships an untested combination. The shorthand form places no such restriction.

Version selection precedence, highest first:

--server-version   >   the model's server_version:   >   the runtime's default_version

Select at deploy time with --server <engine> --server-version <v>, and run variants side by side with --as <alt-name>. See Deploy a Model.

The engines themselves — images, available versions, and shared settings — are defined once under runtimes:. See Runtimes.

Chat templates

chat_template accepts either inline Jinja or an absolute path.

Inline Jinja (use a | block scalar for multi-line) is stored in a <model>-chat-template ConfigMap and mounted read-only at /etc/chat-template. It is deliberately not inlined into the LLMInferenceService, because the KServe controller runs the spec through Go text/template and rejects {{ ... }}. The ConfigMap is created on deploy and removed on undeploy.

An absolute path (leading /) is passed through untouched and must already exist in the pod — for example a file you placed on the model PVC, visible at /mnt/models/<file>. vLLM exits if it is missing, so prefer inline Jinja unless the file is genuinely external.

Auto-deploy on install

By default no models are deployed during install — the inference layer only stands up serving infrastructure. Bringing a model up automatically takes two settings:

1. Flag the model in the catalog:

   yaml Copy

   - name: qwen3-0-6b
     model_id: Qwen/Qwen3-0.6B
     autodeploy: true

2. Enable the auto-deploy phase in this repo's config.yaml, which ships disabled:

   yaml Copy

   llm_services_deploy_models: true

Once enabled, only models flagged autodeploy: true are deployed.

Auto-deploy is fault-tolerant. If a flagged model fails — a gated model with no HF_TOKEN, a transient download error — the install does not abort. The serving infrastructure is already up, the failure is reported in a summary, and the model can be deployed manually afterward. Set model_deploy_strict: true to make any auto-deploy failure fatal instead.

To skip the deploy phase for a single run without editing config, use ./es_auto_installer.sh install inference --skip-models (or SKIP_MODEL_DEPLOYMENT=true).

Extra flags can be appended to every auto-deploy invocation via model_deploy_flags, e.g. "--wait-timeout 1800" for large models.