← Docs Index

Soft affinity-based workload placement for multi-node Intel® AI for Enterprise Solutions clusters, separating platform components from inference workloads without hard enforcement.

Overview

When node_topology_enabled: true (default) in multi-node deployments:

• Platform components (Keycloak, PostgreSQL, Envoy Gateway, cert-manager, MetalLB, KServe controller, AI Gateway controller) prefer control-plane nodes
• Inference workloads (vLLM pods, model serving) prefer worker nodes
• Scheduler uses soft preferences — can override if resources constrained
• No taints — flexible scheduling, no hard failures

Configuration

Enable/Disable

# env/<name>/global_config.yaml
node_topology_enabled: true  # default: true

Set to false to disable workload separation (all pods can land anywhere).

Behavior by Deployment Mode

How It Works

1. Node Labeling (Kubernetes Role)

Multi-node clusters get labeled during Kubernetes provisioning:

# Control-plane nodes
kubectl label node master1 workload-class=platform

# Worker nodes  
kubectl label node worker1 workload-class=inference
kubectl label node worker2 workload-class=inference

File: roles/kubernetes/tasks/apply_node_topology.yaml

2. Pod Affinity (Component Roles)

Affinity rules are defined once in playbooks/includes/preflight.yaml as shared facts (_platform_affinity, _inference_affinity) and referenced by every component. This keeps the placement policy in a single source of truth — changing the rule updates every component.

When node_topology_enabled: false, the facts resolve to {} and Helm charts receive empty affinity (no effect).

Platform Components (prefer control-plane)

All platform Helm charts reference _platform_affinity:

# Example: roles/metallb/tasks/install.yaml
values:
  controller:
    affinity: "{{ _platform_affinity | default({}) }}"

The fact expands to:

nodeAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
          - key: workload-class
            operator: In
            values: [platform]

Affected components:

• Keycloak (StatefulSet) (roles/keycloak/templates/keycloak.yaml.j2)
• PostgreSQL (CNPG Cluster) (roles/postgresql/templates/cnpg-cluster.yaml.j2)
• cert-manager (controller + webhook + cainjector) (roles/cert_manager/tasks/install.yaml)
• MetalLB controller (roles/metallb/tasks/install.yaml)
• Envoy Gateway controller + data plane (roles/envoy_gateway/tasks/install.yaml)
• Envoy AI Gateway controller + data plane (ext/enterprise.ai-inference/roles/envoy_ai_gateway/tasks/install.yaml)
• KServe controller (ext/enterprise.ai-inference/roles/kserve/tasks/install.yaml)

Envoy data-plane pinning: The actual Envoy proxy pods (which handle external HTTPS traffic, not just the controller) are pinned via an EnvoyProxy CRD referenced from the GatewayClass — see eg-proxy-config / ai-gateway-proxy-config.

Inference Workloads (prefer workers)

Inference pods include dual preference (workload-class + fallback):

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
            - key: workload-class
              operator: In
              values: [inference]
      - weight: 90
        preference:
          matchExpressions:
            - key: node-role.kubernetes.io/control-plane
              operator: DoesNotExist

Affected templates (static YAML — processed by model_manager at runtime, not Ansible):

• LLMInferenceService (ext/enterprise.ai-inference/model_manager/templates/llm-inference-service.yaml)
• InferenceService (ext/enterprise.ai-inference/model_manager/templates/inference-service.yaml)

Benefits

Resource Isolation

• Platform components don't compete with inference for CPU/memory
• Inference workloads get dedicated worker resources

NUMA Optimization

• Workers can be dedicated to model serving (NUMA affinity, huge pages)
• Control-plane nodes stable for cluster management

Control Plane Stability

• Masters not overloaded by inference pods
• Cluster operations remain responsive

Flexible Scheduling

• Soft preferences allow scheduler overrides if needed
• No hard failures from resource constraints
• Works with node maintenance (drain/cordon)

Verification

Check Node Labels

kubectl get nodes --show-labels | grep workload-class

Expected output (multi-node):

master1   Ready   control-plane   ...   workload-class=platform
worker1   Ready   <none>          ...   workload-class=inference
worker2   Ready   <none>          ...   workload-class=inference

Check Pod Placement

# Platform components (should be on control-plane)
kubectl get pods -n keycloak -o wide
kubectl get pods -n postgresql -o wide
kubectl get pods -n envoy-gateway-system -o wide

# Inference workloads (should be on workers)
kubectl get pods -n llm-services -o wide
kubectl get pods -n inference-services -o wide

Verify Affinity Rules

# Check platform component affinity
kubectl get deployment keycloak-operator -n keycloak -o yaml | grep -A 10 affinity

# Check inference pod affinity
kubectl get pod <inference-pod-name> -n llm-services -o yaml | grep -A 15 affinity

Troubleshooting

Pods Not Landing on Preferred Nodes

Cause: Soft affinity is a preference, not a requirement. Scheduler may override if:

• Preferred nodes lack resources (CPU/memory)
• Node is cordoned/drained
• Other constraints (PVC affinity, topology spread)

Resolution: Check node capacity and pod resource requests:

kubectl top nodes
kubectl describe node <node-name>

Existing Cluster (BYO Mode)

If using an existing cluster, labels are not applied automatically. Label manually:

# Label control-plane nodes
kubectl label nodes master1 master2 master3 workload-class=platform

# Label worker nodes
kubectl label nodes worker1 worker2 worker3 workload-class=inference

Then redeploy components for affinity to take effect:

./es_auto_installer.sh teardown platform --env <name>
./es_auto_installer.sh install platform --env <name>

Disable Topology (Emergency Override)

If soft affinity causes issues:

# Disable in config
vim env/<name>/global_config.yaml  # set node_topology_enabled: false

# Redeploy affected layers
./es_auto_installer.sh teardown --all --env <name>
./es_auto_installer.sh install --all --env <name>

Or remove labels manually:

kubectl label nodes --all workload-class-

Architecture Notes

Why Does Intel® AI for Enterprise Solutions Use Soft Affinity Instead of Taints?

Taints enforce hard placement rules (NoSchedule, NoExecute). Rejected because:

• Hard failures if resources constrained
• Breaks existing workloads without tolerations
• Complicates DaemonSet deployments (CNI, monitoring)
• Inflexible for mixed workloads

Soft affinity balances separation with flexibility — scheduler respects preferences but can override when needed.

Weight Values

• 100: Primary preference (workload-class match)
• 90: Fallback preference (avoid control-plane for inference)

Higher weight = stronger preference. Scheduler sums weights across all rules.

Label Stability

Labels are applied idempotently — re-running installer won't duplicate or conflict. Safe to re-apply after node addition/replacement.

Related Docs

External References

• Kubernetes Node Affinity — official docs on the preferredDuringSchedulingIgnoredDuringExecution rules this feature uses
• Taints and Tolerations — the hard-enforcement alternative this design deliberately avoids (see Why Does Intel® AI for Enterprise Solutions Use Soft Affinity Instead of Taints? above)