← Docs Index

Everything you need before deploying Intel AI for Enterprise RAG on top of the Intel® AI for Enterprise Solutions platform.

Platform prerequisites

Intel AI for Enterprise RAG is deployed as the erag layer on top of the platform. Enterprise AI Solutions provisions Kubernetes, storage, observability, model serving, and authentication before RAG can be deployed.

Hardware requirements for RAG

These are additional requirements on top of the platform layer. Sizing below assumes the platform and model-serving layers are already running.

Xeon-only deployment

To deploy Intel AI for Enterprise RAG using Xeon only:

The limited single-user deployment (32 cores / 64 GB RAM) is suitable only for evaluation, development, or single-user testing. It requires tuning model resource limits and disabling autoscaling. See Deploying on Minimum Hardware below.

Software prerequisites

• Hugging Face model access: Ensure you have the necessary access to download and use the models your deployment will serve. Gated models (Llama, Mistral, Gemma) require a Hugging Face token exported as HF_TOKEN before install.

• Multi-node clusters require ReadWriteMany (RWX) storage: If deploying across multiple nodes, a CSI driver supporting ReadWriteMany access mode is required. Enterprise AI Solutions can provision NFS (storage_backend: nfs) or NetApp Trident (storage_backend: netapp-trident). Single-node deployments use local-path by default. See Platform Storage Documentation for configuration.

Cluster provisioning and storage

Cluster provisioning, operating system requirements, Kubernetes versions, storage backends, and node topology are owned by the Enterprise AI Solutions, not by the RAG layer.

For those topics, see:

• Platform Prerequisites - OS, tooling, hardware minimums for the platform
• Platform Quickstart - installing the cluster
• Multi-Node & BYO Cluster - multi-node and bring-your-own-Kubernetes setups
• Storage Configuration - NFS, Ceph, NetApp Trident

Deploying on Minimum Hardware

When deploying on the minimum hardware configuration (60 logical cores / 128 GB RAM) or the limited single-user configuration (32 logical cores / 64 GB RAM), additional tuning is required.

KV cache reduction (required)

Model servers cache key-value pairs during inference. The default cache size (10 GB) must be reduced to 1 GB for minimum-hardware deployments.

Edit the model catalog after running init erag (it creates the catalog at env/<name>/models-rag.yaml):

nano env/<name>/models-rag.yaml

Find the models you will deploy and reduce their VLLM_CPU_KVCACHE_SPACE setting from "10" to "1":

  - name: llama3-8b-awq
    model_id: casperhansen/llama-3-8b-instruct-awq
    category: llm
    runtime: vllm
    server_version: "0.24.0"
    cpu: 32
    memory: 24Gi
    env:
      VLLM_CPU_KVCACHE_SPACE: "1"    # reduced from "10"
    args: [--max-num-seqs, "32", ...]

Repeat for every model in inference_models (LLM, embedding, reranking).

Limited single-user (32 cores / 64 GB) additional steps

For the 32 logical cores / 64 GB RAM configuration, also set the following in env/<name>/config.erag.yaml before running install erag:

hpa_enabled: false                  # disable Horizontal Pod Autoscaling
kubernetes_cpu_policy: best-effort  # disable NRI CPU pinning

Additionally, reduce the model server CPU and memory limits in env/<name>/models-rag.yaml:

  - name: llama3-8b-awq
    cpu: 16                          # reduced from 32
    memory: 16Gi                     # reduced from 24Gi
    replicas: 1                      # single replica only
    env:
      VLLM_CPU_KVCACHE_SPACE: "1"

Next steps

Once prerequisites are met, proceed to Quickstart to deploy Intel AI for Enterprise RAG.