Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Automatically scale Distributed Inference with llm-d model deployments

You can configure intelligent, inference-aware automatic scaling for Distributed Inference with llm-d model deployments using the workload variant autoscaler (WVA). WVA automatically adjusts replica counts and AI accelerator allocation based on real-time traffic signals and hardware capacity, helping you optimize infrastructure usage while maintaining inference performance.

Important

Workload variant autoscaler is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

7.1. Workload variant autoscaler overview
Copy link

The workload variant autoscaler (WVA) is a Kubernetes controller that provides intelligent, inference-aware autoscaling for Distributed Inference with llm-d model deployments. You can use the WVA to automatically adjust replica counts and AI accelerator allocation based on real-time traffic signals, hardware capacity, and workload characteristics instead of relying on generic metrics such as CPU or memory utilization.

Traditional Kubernetes autoscaling approaches rely on metrics such as CPU utilization, memory consumption, or queries per second (QPS). These metrics do not accurately represent the resource demands of large language model (LLM) inference workloads because LLM inference is GPU-bound, not CPU-bound. CPU utilization does not reflect the actual load on the inference server.

The WVA addresses these limitations by deriving scaling decisions from inference-specific signals that directly reflect the state of the model server.

The WVA operates as part of the Distributed Inference with llm-d inference stack alongside the following components:

vLLM model servers
The inference serving backends that the WVA scales.
Inference Gateway
Routes incoming requests to available model server replicas.
Inference scheduler
Selects optimal endpoints for each request based on load and locality.
KV cache management
Manages key-value cache storage and transfer between replicas. This component is embedded in the inference scheduler and does not run as a separate container.
KEDA
Provides the autoscaling actuator that the WVA uses to manage HPA resources. KEDA reads the wva_desired_replicas metric from Prometheus and manages the HPA resource that scales the target deployment.

7.2. Key metrics for autoscaling decisions
Copy link

The workload variant autoscaler (WVA) uses the following inference-specific metrics from vLLM model servers to drive scaling decisions:

  • KV cache utilization (kserve_vllm:kv_cache_usage_perc): The percentage of key-value cache in use on each replica. The WVA queries the maximum value over a 1-minute window. A replica is considered saturated when its KV cache utilization reaches the configured kvCacheThreshold.
  • Queue length (kserve_vllm:num_requests_waiting): The number of requests waiting to be processed on each replica. The WVA queries the maximum value over a 1-minute window. A replica is considered saturated when its queue length reaches the configured queueLengthThreshold.

The WVA evaluates these metrics across all replicas of a deployment to calculate spare capacity and determine whether to scale up or scale down.

7.3. How the workload variant autoscaler works
Copy link

The workload variant autoscaler (WVA) is deployed as a controller that monitors VariantAutoscaling custom resources and adjusts replica counts for model server deployments. To deploy the WVA controller, you must enable it in the DataScienceCluster custom resource.

There is a single inference stack per namespace. You cannot deploy multiple inference stacks in the same namespace.

You can create multiple LLMInferenceService CRs within a single stack to serve the same model on different accelerator types. For example, you might have one deployment on H100 GPUs and another on A100 GPUs. The WVA coordinates scaling across all variants that share the same model identifier, scaling up the cheapest variant first.

The WVA uses a saturation-based spare capacity model to make scaling decisions. It evaluates three main dimensions:

Token-level demand estimation
By measuring KV cache utilization as a proxy for tokens actively being processed, and the number of requests waiting in queue, the WVA computes token-level demand rather than relying on generic throughput metrics such as CPU or memory utilization.
Saturation-aware spare capacity
The WVA compares token-level demand against each replica’s effective capacity: the minimum of its memory-bound KV cache size limit and compute-bound limit, or batch saturation. Scaling decisions are triggered when spare capacity across replicas falls below a threshold, rather than waiting for replicas to become fully saturated.
Hardware-aware optimization

At a per-variant level, the WVA accounts for AI accelerator saturation levels. In the LLMISVC CRD, different accelerators can be selected for the requirements of each variant, for example:

  • Prefill optimized
  • Generic inference workers capable of both workloads
  • Other configurations that can change the model server’s behavior.

7.3.1. Saturation scaling algorithm
Copy link

The workload variant autoscaler (WVA) uses a saturation-based spare capacity model to make scaling decisions. The algorithm evaluates whether the current replica set has sufficient spare capacity to absorb traffic increases.

The scaling algorithm works as follows:

  1. Identify non-saturated replicas: A replica is non-saturated if its KV cache utilization is below kvCacheThreshold and its queue length is below queueLengthThreshold.
  2. Calculate spare capacity: For each non-saturated replica, the WVA calculates the spare KV capacity and spare queue capacity.
  3. Average spare capacity: The WVA averages the spare capacity across all non-saturated replicas.
  4. Scale-up decision: The WVA signals a scale-up if the average spare KV capacity is below kvSpareTrigger or the average spare queue capacity is below queueSpareTrigger.
  5. Scale-down safety check: The WVA simulates removing one replica and redistributing load. Scale-down is safe only if remaining spare capacity still exceeds the triggers and at least 2 non-saturated replicas exist.

A variant is eligible for scaling if it does not already have a pending scaling decision from a previous reconciliation cycle. If a variant’s desired replica count differs from its current count, the WVA preserves the existing decision and skips that variant when selecting candidates for new scaling actions.

To prevent over-provisioning during the model loading period, the WVA skips variants that have pending replicas when selecting a variant to scale up. A replica is considered pending when the pod exists but is not yet reporting metrics. This cascade scaling prevention ensures that only variants with all replicas ready are eligible for additional scale-up.

You can enable intelligent autoscaling for your Distributed Inference with llm-d model deployments by configuring the workload variant autoscaler (WVA). The WVA controller is automatically deployed when the OpenShift AI Operator is installed. After you configure autoscaling in the LLMInferenceService custom resource, the WVA automatically adjusts the replica count of your model server based on real-time inference traffic and AI accelerator capacity.

Prerequisites

  • You have an OpenShift cluster on version 4.20 or later.
  • You have installed the OpenShift CLI (oc).
  • You have logged in as a user with cluster-admin privileges.

  • You have installed Red Hat OpenShift AI 3.4 with the Distributed Inference with llm-d stack enabled.

    Important

    Dependencies for installing and using the Distributed Inference with llm-d stack require configuration of the underlying Red Hat OpenShift AI deployment. You cannot use WVA without first configuring OpenShift AI.

  • Optional: You have installed jq.
  • You have installed compatible AI accelerators in the cluster.
  • You have enabled OpenShift User Workload Monitoring (UWM) in the cluster for Prometheus metrics collection. See Configuring user workload monitoring.
  • You have installed the custom-metrics-autoscaler Operator from OperatorHub. See Automatically scaling pods with the Custom Metrics Autoscaler Operator.
  • You have installed the Red Hat Connectivity Link Operator from OperatorHub. See Installing Red Hat Connectivity Link.

  • You have installed the Red Hat OpenShift Service Mesh 3 Operator in your cluster.

    Note

    The Operator is installed by default on any OpenShift cluster version 4.20 or later.

  • DataScienceClusterInitialization (DSCI) and DataScienceCluster (DSC) CRDs exist in your cluster, enabling the workload-variant-autoscaler-controller-manager, llmisvc-controller-manager and kserve-controller-manager controllers. The DataScienceClusterInitialization controller is automatically created by the OpenShift AI Operator. This example excerpt from the DataScienceCluster manifest enables the WVA controller: 

    apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: default-dsc
  labels:
    app.kubernetes.io/name: datasciencecluster
spec:
  components:
    kserve:
      # Create the KServe and LLMISVC controller managers
      managementState: "Managed"
      nim:
        managementState: "Managed"
      rawDeploymentServiceConfig: "Headed"
      wva:
        # Create the workload-variant-autoscaler controller manager
        managementState: "Managed"
        # ...Other DSC components as desired
  • You have confirmed that no other LLMInferenceService CR exists in the namespace you intend to deploy the WVA in. Each namespace should contain a single Distributed Inference with llm-d inference stack only.

Procedure

  1. Verify that the required controllers exist in the cluster. Run the following command: 

    $ oc get pods -n redhat-ods-applications \
  -l 'app.kubernetes.io/name in (kserve-controller-manager, llmisvc-controller-manager, workload-variant-autoscaler)'

    All controllers should report as ready: 

    NAME                                            READY   STATUS    RESTARTS   AGE
kserve-controller-manager                       1/1     Running   0          38m
llmisvc-controller-manager                      1/1     Running   0          38m
workload-variant-autoscaler-controller-manager  1/1     Running   0          38m

  2. Verify the default scaling configuration.

    Run the following command to inspect the workload-variant-autoscaler-saturation-scaling-config ConfigMap: 

    $ oc get cm workload-variant-autoscaler-saturation-scaling-config -n redhat-ods-applications -o jsonpath='{.data.default}'

    You should see the following default values: 

    kvCacheThreshold: 0.80
queueLengthThreshold: 5
kvSpareTrigger: 0.1
queueSpareTrigger: 3
enableLimiter: false

    These values represent the default scaling thresholds that the WVA uses to determine when to scale your inference workloads up or down.

  3. Grant the KEDA controller permissions to read from OpenShift monitoring resources.

    Note

    The Custom Metrics Autoscaler Operator does not automatically have permission to read metrics from the openshift-monitoring or openshift-user-workload-monitoring stack because OpenShift restricts access to cluster monitoring APIs. Access to Prometheus and Thanos endpoints requires explicit RBAC permissions such as the cluster-monitoring-view role.

    1. Authorize KEDA to read OpenShift user workload monitoring (UWM) metrics. Create the following CRs: 

      apiVersion: v1
kind: ServiceAccount
metadata:
  name: keda-metrics-reader
  namespace: openshift-keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-metrics-reader-monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-monitoring-view
subjects:
  - kind: ServiceAccount
    name: keda-metrics-reader
    namespace: openshift-keda
---
apiVersion: v1
kind: Secret
metadata:
  name: keda-metrics-reader-token
  namespace: openshift-keda
  annotations:
    kubernetes.io/service-account.name: keda-metrics-reader
type: kubernetes.io/service-account-token

    2. Allow KEDA to authenticate against OpenShift user workload monitoring. To give KEDA the proper permissions, create a ClusterTriggerAuthentication CR, which references the keda-metrics-reader-token and Secret CRs acting as a Token for the keda-metrics-reader and ServiceAccount CRs created in the previous step. Create the ClusterTriggerAuthentication CR: 

      apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: ai-inference-keda-thanos
spec:
  secretTargetRef:
    - parameter: bearerToken
      name: keda-metrics-reader-token
      key: token
    - parameter: ca
      name: keda-metrics-reader-token
      key: ca.crt

      KEDA can now pull metrics from the user workload monitoring stack.

  4. Create an inference stack with autoscaling enabled.

    1. Create the namespace.

      Because only one inference stack per namespace is supported, create a dedicated namespace for the deployment. You can use any namespace, provided you adjust the manifests accordingly. 

      $ oc create ns autoscaling-example && oc project autoscaling-example

    2. Create the gateway.

      Create a gateway that uses TLS with OpenShift Service Mesh. OpenShift Service Mesh is installed by default on OpenShift version 4.20 and later. To create the Gateway, apply the following ConfigMap and Gateway manifests: 

      apiVersion: v1
kind: ConfigMap
metadata:
  name: autoscaling-example-gateway-config
  namespace: autoscaling-example
data:
  service: |
    metadata:
      annotations:
        service.beta.openshift.io/serving-cert-secret-name: "autoscaling-example-gateway-tls"
    spec:
      type: ClusterIP
  deployment: |
    spec:
      template:
        spec:
          containers:
            - name: istio-proxy
              resources:
                limits:
                  cpu: "16"
                  memory: 16Gi
                requests:
                  cpu: "4"
                  memory: 4Gi

      +

      Note

      This example uses a ClusterIP service type for compatibility across OpenShift environments that do not have LoadBalancer service type integration. The istio-proxy resource limits are increased to handle the significant load generation required to trigger a scaling event during verification. Adjust these values according to your expected traffic patterns. 

      apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: autoscaling-example-gateway
  namespace: autoscaling-example
spec:
  gatewayClassName: data-science-gateway-class
  infrastructure:
    parametersRef:
      group: ""
      kind: ConfigMap
      name: autoscaling-example-gateway-config
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: autoscaling-example-gateway-tls
      mode: Terminate

    3. Create the LLMInferenceService CR with autoscaling configurations enabled. 

      apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: autoscaling-example-llama
  namespace: autoscaling-example
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
    prometheus.io/path: "/metrics"
    security.opendatahub.io/enable-auth: 'false'
spec:
  router:
    scheduler: {}
    route: {}
    gateway:
      refs:
      - name: autoscaling-example-gateway
        namespace: autoscaling-example
  model:
    uri: hf://Qwen/Qwen2.5-7B-Instruct
    name: Qwen/Qwen2.5-7B-Instruct
  labels:
    inference.optimization/acceleratorName: H100
  scaling:
    minReplicas: 1
    maxReplicas: 5
    wva:
      keda:
        pollingInterval: 5
        cooldownPeriod: 30
  template:
    containers:
      - name: main
        image: registry.redhat.io/rhaii/vllm-cuda-rhel9:3.4.0
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: 1
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: 1
        startupProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS

      The following fields are important to understand when you customize the LLMInferenceService CR for your use case:

      • .metadata.annotations.security.opendatahub.io/enable-auth: Disables authentication and rate limiting for this example. In OpenShift AI, user authentication and rate limiting are handled by the Red Hat Connectivity Link Operator. This example disables authentication so that the load test can generate sufficient traffic to trigger a scaling event without being rate-limited. Production deployments should not set this annotation.
      • .spec.gateway.refs: Specifies the reference to the pre-created Gateway that the LLMInferenceService uses.
      • .spec.labels.inference.optimization/acceleratorName: Specifies the type of accelerator this workload uses, such as A100, H100, or cpu. The WVA uses this value to group variants by hardware type and look up accelerator-specific performance data when making scaling decisions. There is no fixed list of valid values: use whatever matches your hardware for semantic identification. If omitted, it defaults to unknown.
      • .spec.scaling: Specifies the user-facing scaling configurations at a per-LLMInferenceService level, which represents a per-inference-stack level. Do not confuse this with the configurations from the workload-variant-autoscaler-saturation-scaling-config ConfigMap, which represents the default scaling configurations that the workload-variant-autoscaler uses to determine the threshold at which it scales your inference workloads up or down.
      • .spec.template.containers[0].image: Specifies the inference image. Different inference images are used for different accelerators. If you are using accelerators, use the corresponding inference image.

      • .spec.template.containers[0].resources: Specifies the resource requests and limits. Adjust the resources as needed for more GPUs and add the tensor parallel flags to the VLLM_ADDITIONAL_ARGS environment variable, for example --tensor-parallel 2.

        Note

        The same requirement applies for accelerated networking devices like RoCE or InfiniBand. Autoscaling inter-node workloads through LeaderWorkerSet is not supported.

      • .spec.template.containers[0].startupProbe, .spec.template.containers[0].readinessProbe, .spec.template.containers[0].livenessProbe: Specifies the probe configurations. Because this example uses TLS end to end, ensure that your probes use the HTTPS scheme. If the probes do not use the TLS scheme, they fail to reach the endpoints and declare the pods as not ready or not healthy.

    4. Create a PrometheusRule to alias inference image metric names.

      By default, vLLM exposes metrics to the vllm namespace, but Red Hat AI Inference remaps these metrics to the kserve_vllm namespace. The WVA only looks for the kserve_vllm-namespaced instances of these metrics. To enable the WVA to pick up these metrics, create the following PrometheusRule: 

      apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vllm-metrics-alias
  namespace: autoscaling-example
  labels:
    monitoring.opendatahub.io/scrape: "true"
spec:
  groups:
  - name: vllm-metric-aliases
    interval: 15s
    rules:
    - record: vllm:kv_cache_usage_perc
      expr: kserve_vllm:kv_cache_usage_perc
    - record: vllm:num_requests_waiting
      expr: kserve_vllm:num_requests_waiting
    - record: vllm:num_requests_running
      expr: kserve_vllm:num_requests_running
    - record: vllm:cache_config_info
      expr: kserve_vllm:cache_config_info
    - record: vllm:request_success_total
      expr: kserve_vllm:request_success_total
    - record: vllm:request_generation_tokens_sum
      expr: kserve_vllm:request_generation_tokens_sum
    - record: vllm:request_generation_tokens_count
      expr: kserve_vllm:request_generation_tokens_count
    - record: vllm:request_prompt_tokens_sum
      expr: kserve_vllm:request_prompt_tokens_sum
    - record: vllm:request_prompt_tokens_count
      expr: kserve_vllm:request_prompt_tokens_count
    - record: vllm:time_to_first_token_seconds_sum
      expr: kserve_vllm:time_to_first_token_seconds_sum
    - record: vllm:time_to_first_token_seconds_count
      expr: kserve_vllm:time_to_first_token_seconds_count
    - record: vllm:time_per_output_token_seconds_sum
      expr: kserve_vllm:time_per_output_token_seconds_sum
    - record: vllm:time_per_output_token_seconds_count
      expr: kserve_vllm:time_per_output_token_seconds_count
    - record: vllm:prefix_cache_hits
      expr: kserve_vllm:prefix_cache_hits
    - record: vllm:prefix_cache_queries
      expr: kserve_vllm:prefix_cache_queries

    5. Verify the LLMInferenceService and ScaledObject CRs were created successfully. 

      $ oc get llmisvc -n autoscaling-example
       
      NAME                        URL   READY   REASON   AGE
autoscaling-example-llama         True             102s
       
      $ oc get scaledobject -n autoscaling-example
       
      NAME                                    SCALETARGETKIND      SCALETARGETNAME                    MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED   TRIGGERS     AUTHENTICATIONS            AGE
autoscaling-example-llama-kserve-keda   apps/v1.Deployment   autoscaling-example-llama-kserve   1     5     True    True     False      False    prometheus   ai-inference-keda-thanos   119s

Verification

Verify autoscaling behavior by confirming that inference server metrics appear in Prometheus, that the WVA emits scaling metrics, and that an end-to-end autoscaling event occurs.

  1. Verify that inference server component metrics appear in Prometheus.

    Ensure that the inputs to the WVA appear in Prometheus, specifically the inference server metrics: 

    $ TOKEN=$(oc whoami -t)
     
    $ THANOS=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}')
for m in num_requests_running num_requests_waiting kv_cache_usage_perc; do
    curl -sk -G -H "Authorization: Bearer $TOKEN" "https://$THANOS/api/v1/query" \
      --data-urlencode "query=vllm:${m}{namespace=\"autoscaling-example\"}" | \
      jq '.data.result[0]'
done

    You should see three separate JSON payloads, one for each metric. The following example shows the expected output format: 

    {
  "metric": {
    "__name__": "vllm:num_requests_running",
    "container": "main",
    "endpoint": "8000",
    "engine": "0",
    "instance": "10.129.0.66:8000",
    "job": "autoscaling-example/kserve-llm-isvc-vllm-engine",
    "llm_isvc_component": "workload",
    "llm_isvc_name": "autoscaling-example-llama",
    "llm_isvc_role": "both",
    "model_name": "Qwen/Qwen2.5-7B-Instruct",
    "namespace": "autoscaling-example",
    "pod": "autoscaling-example-llama-kserve-649dc95869-ntlwq",
    "prometheus": "openshift-user-workload-monitoring/user-workload"
  },
  "value": [
    1774817937.592,
    "0"
  ]
}
    Note

    The WVA does not currently use metrics from the scheduler.

  2. Verify that the WVA is emitting metrics back to Prometheus.

    The metrics to look for are wva_current_replicas, wva_desired_replicas, and wva_desired_ratio. Run the following commands to query these metrics: 

    $ TOKEN=$(oc whoami -t)
     
    $ THANOS=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}')
for m in wva_desired_replicas wva_current_replicas wva_desired_ratio; do
  curl -sk -G -H "Authorization: Bearer $TOKEN" "https://$THANOS/api/v1/query" \
    --data-urlencode "query=${m}{exported_namespace=\"autoscaling-example\"}" \
    | jq '.data.result[0]'
done

    You should see JSON payloads for each metric. The following example shows the expected output format: 

    {
  "metric": {
    "__name__": "wva_desired_replicas",
    "accelerator_type": "H100",
    "endpoint": "https",
    "exported_namespace": "autoscaling-example",
    "instance": "10.128.0.64:8443",
    "job": "workload-variant-autoscaler-controller-manager-metrics-service",
    "namespace": "redhat-ods-applications",
    "pod": "workload-variant-autoscaler-controller-manager-85b8d895cd-x6gnr",
    "prometheus": "openshift-user-workload-monitoring/user-workload",
    "service": "workload-variant-autoscaler-controller-manager-metrics-service",
    "variant_name": "autoscaling-example-llama-kserve-va"
  },
  "value": [
    1774819127.020,
    "1"
  ]
}

    At this point, verify the following resources that were created from your LLMInferenceService: 

    $ oc get va autoscaling-example-llama-kserve-va -n autoscaling-example
     
    NAME                                  TARGET                             MODEL                      OPTIMIZED   METRICSREADY   AGE
autoscaling-example-llama-kserve-va   autoscaling-example-llama-kserve   Qwen/Qwen2.5-7B-Instruct   1           True           81m
     
    $ oc get scaledObject -n autoscaling-example
     
    NAME                                    SCALETARGETKIND      SCALETARGETNAME                    MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED   TRIGGERS     AUTHENTICATIONS            AGE
autoscaling-example-llama-kserve-keda   apps/v1.Deployment   autoscaling-example-llama-kserve   1     5     True    True     Unknown    False    prometheus   ai-inference-keda-thanos   83m

    The VariantAutoscaling CR should show METRICSREADY=True and the ScaledObject should show READY=true. Once the WVA controller starts reconciling the ScaledObject, it should also become ACTIVE=True. If you do not see these values, run oc describe on these resources and check their Status sections.

  3. Verify that the inference worker gets autoscaled.

    To make scaling events easier to trigger during testing, set the kvCacheThreshold key in the workload-variant-autoscaler-saturation-scaling-config ConfigMap in the redhat-ods-applications namespace to 0.10: 

    $ oc edit cm workload-variant-autoscaler-saturation-scaling-config -n redhat-ods-applications

    Change kvCacheThreshold from 0.80 to 0.10.

    The following script deploys a load-generator pod inside the cluster and watches for scaling events. The script resolves the gateway service, sends thousands of concurrent requests with long token generation to saturate the vLLM KV cache past the WVA scaling threshold, and monitors the replica count every 5 seconds:

    Note

    The scaling behavior and load values in this script are dependent on the accelerators in use. This example was tested with NVIDIA H100 AI accelerators. 

    #!/bin/bash
set -euo pipefail

NS="${1:-autoscaling-example}"
ISVC="${2:-autoscaling-example-llama}"
CONCURRENCY="${3:-200}"
REQUESTS="${4:-5000}"
POD_NAME="load-test-$(date +%s)"
CM_NAME="script-${POD_NAME}"

GATEWAY_SVC=$(oc get svc -n "$NS" -l gateway.networking.k8s.io/gateway-name \
  -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "$GATEWAY_SVC" ]; then
  echo "ERROR: No gateway service found in namespace $NS"
  exit 1
fi
URL="https://${GATEWAY_SVC}.${NS}.svc.cluster.local/${NS}/${ISVC}/v1/chat/completions"

echo "=== WVA Autoscaling Test ==="
echo "URL:         $URL"
echo "Concurrency: $CONCURRENCY"
echo "Requests:    $REQUESTS"
echo ""
oc get deployment "${ISVC}-kserve" -n "$NS" \
  -o jsonpath='Initial state: {.spec.replicas}/{.status.readyReplicas} ready' && echo ""
echo ""

cleanup() {
  echo ""
  echo "Cleaning up..."
  oc delete pod "$POD_NAME" -n "$NS" --ignore-not-found --wait=false 2>/dev/null
  oc delete configmap "$CM_NAME" -n "$NS" --ignore-not-found 2>/dev/null
}
trap cleanup EXIT INT TERM

read -r -d '' LOAD_SCRIPT << 'EOF' || true
#!/bin/sh
set -e
URL="$1"; CONCURRENCY="$2"; REQUESTS="$3"

cat > /tmp/req.sh << 'REQEOF'
#!/bin/sh
curl -sk --max-time 600 "$1" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"Qwen/Qwen2.5-7B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Request $2. Write a detailed essay about topic $2 covering history, analysis, and predictions.\"}],\"max_tokens\":2048}" \
  -o /dev/null -w "req=$2 status=%{http_code} time=%{time_total}s\n"
REQEOF
chmod +x /tmp/req.sh

echo "Smoke test..."
STATUS=$(curl -sk --max-time 30 "$URL" \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Hi"}],"max_tokens":5}' \
  -o /dev/null -w "%{http_code}")
echo "Status: $STATUS"
if [ "$STATUS" != "200" ]; then
  echo "ERROR: Smoke test failed (HTTP $STATUS)"
  exit 1
fi

echo "Sending $REQUESTS requests ($CONCURRENCY concurrent)..."
seq 1 "$REQUESTS" | xargs -P "$CONCURRENCY" -I{} /tmp/req.sh "$URL" {}
echo "Done."
EOF

oc create configmap "$CM_NAME" -n "$NS" --from-literal=load.sh="$LOAD_SCRIPT"

cat <<MANIFEST | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: $POD_NAME
  namespace: $NS
spec:
  restartPolicy: Never
  containers:
  - name: load
    image: curlimages/curl
    command: ["sh", "/scripts/load.sh"]
    args: ["$URL", "$CONCURRENCY", "$REQUESTS"]
    resources:
      requests: { cpu: "4", memory: "4Gi" }
      limits:   { cpu: "8", memory: "8Gi" }
    volumeMounts:
    - { name: script, mountPath: /scripts }
  volumes:
  - name: script
    configMap: { name: $CM_NAME, defaultMode: 0755 }
MANIFEST

echo "Waiting for pod..."
oc wait --for=condition=Ready pod/"$POD_NAME" -n "$NS" --timeout=120s 2>/dev/null || true
sleep 2

POD_PHASE=$(oc get pod "$POD_NAME" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null)
if [ "$POD_PHASE" = "Failed" ]; then
  echo "ERROR: Pod failed:"
  oc logs "$POD_NAME" -n "$NS" 2>/dev/null
  exit 1
fi

oc logs -f "$POD_NAME" -n "$NS" 2>/dev/null &
LOGS_PID=$!

echo ""
echo "--- Watching replicas (Ctrl+C to stop) ---"
for _ in $(seq 1 120); do
  replicas=$(oc get deployment "${ISVC}-kserve" -n "$NS" -o jsonpath='{.spec.replicas}' 2>/dev/null)
  ready=$(oc get deployment "${ISVC}-kserve" -n "$NS" -o jsonpath='{.status.readyReplicas}' 2>/dev/null)
  echo "  [$(date +%H:%M:%S)] Replicas: ${replicas:-?} (${ready:-0} ready)"
  [ "${replicas:-1}" -gt 1 ] && echo "  ** Scale-up detected! **"
  oc get pod "$POD_NAME" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null \
    | grep -q "Succeeded\|Failed" && echo "  Load pod finished." && break
  sleep 5
done

kill "$LOGS_PID" 2>/dev/null || true
wait "$LOGS_PID" 2>/dev/null || true

    You should see the replica count increase as load is applied: 

      [17:30:03] Replicas: 1 (1 ready)
  [17:30:58] Replicas: 2 (1 ready)
  ** Scale-up detected! **

    During the load, you should see additional inference worker pods: 

    $ oc get pods -n autoscaling-example
     
    NAME                                                              READY   STATUS    RESTARTS   AGE
autoscaling-example-gateway-data-science-gateway-class-66c4qvpb   1/1     Running   0          81m
autoscaling-example-llama-kserve-56b64d69d-b297x                  1/1     Running   0          106m
autoscaling-example-llama-kserve-56b64d69d-kvvmc                  1/1     Running   0          10m
autoscaling-example-llama-kserve-56b64d69d-m78nz                  1/1     Running   0          7m1s
autoscaling-example-llama-kserve-router-scheduler-77cd5549p4w95   2/2     Running   0          106m

    After the load ceases, the deployment scales back down to 1 replica: 

    NAME                                                              READY   STATUS    RESTARTS   AGE
autoscaling-example-gateway-data-science-gateway-class-66c4qvpb   1/1     Running   0          81m
autoscaling-example-llama-kserve-56b64d69d-b297x                  1/1     Running   0          106m
autoscaling-example-llama-kserve-router-scheduler-77cd5549p4w95   2/2     Running   0          106m
    Note

    The WVA uses a stair-step methodology to scaling, imposing a stabilization window on scaling events so that a traffic burst does not greedily attempt to occupy all your GPUs. If load continues past the stabilization window, you may see additional scaling events.

7.5. VariantAutoscaling CR configuration reference
Copy link

You configure autoscaling behavior by creating a VariantAutoscaling custom resource (CR).

Important

When multiple configuration sources are available, the workload variant autoscaler applies the following precedence order, from highest to lowest.

  1. CLI flags
  2. Environment variables
  3. ConfigMap values
  4. Built-in defaults

Namespace-scoped ConfigMap resources override cluster-level ConfigMap resources for the same parameters. To enable namespace-scoping, add the wva.llmd.ai/config-enabled: "true" label to the target namespace.

The following is an example VariantAutoscaling custom resource (CR). 

apiVersion: llmd.ai/v1alpha1
kind: VariantAutoscaling
metadata:
  name: llama-70b-premium-h100
  namespace: llm-inference
  labels:
    app: llm-inference
    model: llama-70b
    variant: premium
    accelerator: h100
spec:
  scaleTargetRef:
    kind: Deployment
    name: llama-70b-premium-h100
  modelID: "meta/llama-3.1-70b"
  minReplicas: 1
  maxReplicas: 5
  • metadata.name: Specifies a name for the VariantAutoscaling resource. Use a descriptive name that identifies the model, variant role, and accelerator type.
  • metadata.namespace: Specifies the namespace where the target deployment runs. Each namespace supports only one Distributed Inference with llm-d inference stack.
  • metadata.labels: Specifies optional labels for operational tracking. The workload variant autoscaler (WVA) does not use these labels for scaling decisions, but they are useful for filtering and monitoring. The accelerator type is determined by the deployment’s actual hardware, not by these labels.
  • spec.scaleTargetRef.kind: Required. Specifies the kind of the Kubernetes resource that the WVA scales. Supported value: Deployment.
  • spec.scaleTargetRef.name: Specifies the name of the Kubernetes resource that the WVA scales. Must match the name of an existing resource in the same namespace.
  • spec.modelID: Specifies the OpenAI API-compatible model identifier for the inference workload.
  • spec.minReplicas: Specifies the minimum number of replicas. The WVA does not scale below this value. Must be 1 or greater. Default: 1.
  • spec.maxReplicas: Specifies the maximum number of replicas. The WVA does not scale above this value. Set this value based on your cluster AI accelerator capacity. Default: 2.

7.6. Saturation scaling ConfigMap reference
Copy link

The workload variant autoscaler (WVA) uses a ConfigMap named workload-variant-autoscaler-saturation-scaling-config for saturation-based scaling thresholds.

Each ConfigMap contains two types of entries:

  • A default entry that contains global parameters that apply to all models in the scope (cluster-wide or namespace-scoped)
  • Optional override entries that contain model-specific customizations with arbitrary names, such as granite-override or llama-override

The following example shows a ConfigMap CR with cluster-level defaults and a model-specific override:

The default entry contains global parameters that apply to all models unless a matching override exists. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: workload-variant-autoscaler-saturation-scaling-config
  namespace: redhat-ods-applications
data:
  default: |
    kvCacheThreshold: <kv_cache_threshold>
    queueLengthThreshold: <queue_length_threshold>
    kvSpareTrigger: <kv_spare_trigger>
    queueSpareTrigger: <queue_spare_trigger>
    enableLimiter: <enable_limiter>
  <override_name>: |
    model_id: "<model_id>"
    namespace: "<namespace>"
    kvCacheThreshold: <kv_cache_threshold>
    kvSpareTrigger: <kv_spare_trigger>
  • kvCacheThreshold: Specifies the KV cache utilization threshold as a percentage. A replica is considered saturated if its KV cache utilization is greater than or equal to this value. Valid range: 0.0 to 1.0. Lower values trigger scale-up sooner. Default: 0.80.
  • queueLengthThreshold: Specifies the queue length threshold. A replica is considered saturated if its queue length is greater than or equal to this value. Lower values trigger scale-up sooner. Default: 5.
  • kvSpareTrigger: Specifies the KV cache spare capacity trigger. The WVA signals a scale-up if the average spare KV capacity across non-saturated replicas falls below this value. Valid range: 0.0 to 1.0. Default: 0.10.
  • queueSpareTrigger: Specifies the queue spare capacity trigger. The WVA signals a scale-up if the average spare queue capacity across non-saturated replicas falls below this value. Default: 3.
  • enableLimiter: Enables the GPU limiter to constrain scaling based on available cluster GPU capacity. When true, the WVA limits desired replicas so that deployments do not request more GPUs than are available in the cluster, preventing new replicas from entering a pending state. Default: false.
  • <override_name>: Override entry name, such as llm-d or granite-override. The WVA matches overrides by using the model_id and namespace fields, not the entry name.
  • model_id: Specifies the model identifier that this override applies to, such as Qwen/Qwen3-0.6B or ibm/granite-13b. The WVA uses this value together with namespace to match the override to a specific model deployment.
  • namespace: Specifies the namespace of the model deployment that this override applies to. The WVA uses this value together with model_id to match the override to a specific model deployment.
Note

The WVA resolves configuration by first looking for an override that matches both model_id and namespace. If no matching override is found, the WVA uses the default entry. Any parameters that are not specified in the matched entry receive hardcoded default values.

7.7. Autoscaling metrics
Copy link

The workload variant autoscaler (WVA) exposes the following autoscaling-specific metrics through Prometheus. These metrics reflect the scaling decisions and state of the WVA controller.

Expand
Table 7.1. WVA autoscaling metrics
Metric nameTypeLabelsDescription

wva_current_replicas

gauge

variant_name, namespace, accelerator_type

The current number of replicas managed by the WVA for each VariantAutoscaling resource.

wva_desired_replicas

gauge

variant_name, namespace, accelerator_type

The target number of replicas calculated by the WVA. This is the metric read by KEDA or HPA for actuation.

wva_replica_scaling_total

counter

variant_name, namespace, direction, reason

The total number of replica scaling events triggered by the WVA. The direction label is up or down. The reason label indicates the cause of the scaling event.

wva_desired_ratio

gauge

variant_name, namespace, accelerator_type

The ratio of desired replicas to current replicas, indicating the scaling pressure on the deployment.

The WVA uses the following vLLM inference metrics as input signals for scaling decisions. These metrics are exposed by the vLLM model server, not by the WVA controller.

Expand
Table 7.2. vLLM saturation metrics used by the WVA
Metric nameDescription

kserve_vllm:kv_cache_usage_perc

KV cache utilization as a percentage. A value of 0.0 to 1.0. The WVA uses max_over_time over a 1-minute window to determine saturation.

kserve_vllm:num_requests_waiting

Number of requests waiting to be processed. The WVA uses max_over_time over a 1-minute window to determine queue depth.

Note

The vLLM saturation metrics must include pod, model_name, and namespace labels for the WVA to correlate metrics with the correct VariantAutoscaling resource.

7.8. HPA behavior defaults
Copy link

The workload variant autoscaler (WVA) uses KEDA to manage HPA resources. KEDA configures the following default HPA scaling behavior. These values control the stabilization window and scaling velocity.

Expand
Table 7.3. HPA scaling behavior defaults
ParameterDefaultDescription

scaleUp.stabilizationWindowSeconds

240

The stabilization window for scale-up decisions, in seconds. The HPA waits this long after the last scaling event before scaling up again.

scaleDown.stabilizationWindowSeconds

240

The stabilization window for scale-down decisions, in seconds.

selectPolicy

Max

The policy selection strategy. Max selects the policy that results in the largest change.

policies[0].type

Pods

The scaling policy type.

policies[0].value

10

The maximum number of pods to add or remove per scaling period.

policies[0].periodSeconds

150

The duration of the scaling period, in seconds.

If your workload variant autoscaler (WVA) deployment is not scaling as expected, use the following troubleshooting steps to diagnose and resolve common issues.

  • Check the WVA controller status:

    View WVA controller logs to diagnose issues: 

    $ oc logs -l app.kubernetes.io/name=workload-variant-autoscaler -n redhat-ods-applications

  • Verify VariantAutoscaling resource conditions:

    Check the status conditions on your VariantAutoscaling resources: 

    $ oc describe va <variant_name> -n <namespace>

    Common condition states:

    • MetricsAvailable: True with reason MetricsFound: Prometheus metrics are available
    • MetricsAvailable: False with reason MetricsMissing: No metrics found for the variant. Check that vLLM metrics are being scraped.
    • MetricsAvailable: False with reason PrometheusError: WVA cannot connect to Prometheus. Check Prometheus configuration and network connectivity.
    • OptimizationReady: True with reason OptimizationSucceeded: WVA successfully calculated scaling decisions
    • OptimizationReady: False with reason OptimizationFailed: Scaling calculation failed. Check WVA logs for details.

  • Verify that KEDA resources are functioning correctly:

    Check the status of the KEDA ScaledObject resources in your namespace: 

    $ oc get scaledobjects -n <namespace>

    Check the KEDA operator logs for errors: 

    $ oc logs -l app=keda-operator -n openshift-keda

  • Verify Prometheus metrics are being collected:

    Confirm that vLLM metrics are available in Prometheus: 

    $ oc port-forward -n openshift-monitoring prometheus-k8s-0 9090:9090

    Then query Prometheus at http://localhost:9090 for vllm:kv_cache_usage_perc and vllm:num_requests_waiting.

  • Check the WVA scaling metrics in Prometheus:

    Query the wva_desired_replicas metric to verify the WVA is calculating scaling targets: 

    $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'wva_desired_replicas'

  • Enable user workload monitoring:

    If metrics are not being collected, verify that OpenShift User Workload Monitoring is enabled: 

    $ oc get configmap cluster-monitoring-config -n openshift-monitoring

    If the ConfigMap does not exist, create it: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top