SupportConsoleDevelopersStart a trialContact

Chapter 9. Automatically scale Distributed Inference with llm-d model deployments

You can configure intelligent, inference-aware automatic scaling for Distributed Inference with llm-d model deployments using the workload variant autoscaler (WVA). WVA automatically adjusts replica counts and AI accelerator allocation based on real-time traffic signals and hardware capacity, helping you optimize infrastructure usage while maintaining inference performance.

Important

Workload variant autoscaler is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

9.1. Workload variant autoscaler overview
Copy link

The workload variant autoscaler (WVA) is a Kubernetes controller that provides intelligent, inference-aware autoscaling for Distributed Inference with llm-d model deployments. You can use the WVA to automatically adjust replica counts and AI accelerator allocation based on real-time traffic signals, hardware capacity, and workload characteristics instead of relying on generic metrics such as CPU or memory utilization.

Traditional Kubernetes autoscaling approaches rely on metrics such as CPU utilization, memory consumption, or queries per second (QPS). These metrics do not accurately represent the resource demands of large language model (LLM) inference workloads because LLM inference is GPU-bound, not CPU-bound. CPU utilization does not reflect the actual load on the inference server.

The WVA addresses these limitations by deriving scaling decisions from inference-specific signals that directly reflect the state of the model server.

The WVA operates as part of the Distributed Inference with llm-d inference stack alongside the following components:

vLLM model servers
The inference serving backends that the WVA scales.
Inference Gateway
Routes incoming requests to available model server replicas.
Inference scheduler
Selects optimal endpoints for each request based on load and locality.
KV cache management
Manages key-value cache storage and transfer between replicas. This component is embedded in the inference scheduler and does not run as a separate container.
KEDA
Provides the autoscaling actuator that the WVA uses to manage HPA resources. KEDA reads the wva_desired_replicas metric from Prometheus and manages the HPA resource that scales the target deployment.

9.2. Key metrics for autoscaling decisions
Copy link

The workload variant autoscaler (WVA) uses the following inference-specific metrics from vLLM model servers to drive scaling decisions:

  • KV cache utilization (vllm:kv_cache_usage_perc): The percentage of key-value cache in use on each replica. The WVA queries the maximum value over a 1-minute window. A replica is considered saturated when its KV cache utilization reaches the configured kvCacheThreshold.
  • Queue length (vllm:num_requests_waiting): The number of requests waiting to be processed on each replica. The WVA queries the maximum value over a 1-minute window. A replica is considered saturated when its queue length reaches the configured queueLengthThreshold.

The WVA evaluates these metrics across all replicas of a deployment to calculate spare capacity and determine whether to scale up or scale down.

9.3. How the workload variant autoscaler works
Copy link

The workload variant autoscaler (WVA) is deployed as a controller that monitors VariantAutoscaling custom resources and adjusts replica counts for model server deployments. To deploy the WVA controller, you must enable it in the DataScienceCluster custom resource.

The WVA uses a saturation-based spare capacity model to make scaling decisions. It evaluates three main dimensions:

Token-level demand estimation
By measuring KV cache utilization as a proxy for tokens actively being processed, and the number of requests waiting in queue, the WVA computes token-level demand rather than relying on generic throughput metrics such as CPU or memory utilization.
Saturation-aware spare capacity
The WVA compares token-level demand against each replica’s effective capacity: the minimum of its memory-bound KV cache size limit and compute-bound limit, or batch saturation. Scaling decisions are triggered when spare capacity across replicas falls below a threshold, rather than waiting for replicas to become fully saturated.
Hardware-aware optimization

At a per-variant level, the WVA accounts for AI accelerator saturation levels. In the LLMISVC CRD, different accelerators can be selected for the requirements of each variant, for example:

  • Prefill optimized
  • Generic inference workers capable of both workloads
  • Other configurations that can change the model server’s behavior.

9.3.1. Saturation scaling algorithm
Copy link

The workload variant autoscaler (WVA) uses a saturation-based spare capacity model to make scaling decisions. The algorithm evaluates whether the current replica set has sufficient spare capacity to absorb traffic increases.

The scaling algorithm works as follows:

  1. Identify non-saturated replicas: A replica is non-saturated if its KV cache utilization is below kvCacheThreshold and its queue length is below queueLengthThreshold.
  2. Calculate spare capacity: For each non-saturated replica, the WVA calculates the spare KV capacity and spare queue capacity.
  3. Average spare capacity: The WVA averages the spare capacity across all non-saturated replicas.
  4. Scale-up decision: The WVA signals a scale-up if the average spare KV capacity is below kvSpareTrigger or the average spare queue capacity is below queueSpareTrigger.
  5. Scale-down safety check: The WVA simulates removing one replica and redistributing load. Scale-down is safe only if remaining spare capacity still exceeds the triggers and at least 2 non-saturated replicas exist.

A variant is eligible for scaling if it does not already have a pending scaling decision from a previous reconciliation cycle. If a variant’s desired replica count differs from its current count, the WVA preserves the existing decision and skips that variant when selecting candidates for new scaling actions.

To prevent over-provisioning during the model loading period, the WVA skips variants that have pending replicas when selecting a variant to scale up. A replica is considered pending when the pod exists but is not yet reporting metrics. This cascade scaling prevention ensures that only variants with all replicas ready are eligible for additional scale-up.

The LLMInferenceService custom resource (CR) includes a spec.parallelism section that controls how inference workloads are distributed across GPUs and nodes. The workload variant autoscaler (WVA) supports scaling distributed inference workloads.

The WVA monitors KV cache utilization and queue depth across all replicas in a parallel deployment. When spare capacity falls below the configured thresholds, the WVA signals a scale-up by increasing the Deployment replica count. Each new replica joins the data-parallel group, increasing the aggregate inference throughput.

For parallel deployments, the WVA accounts for the following factors:

  • The accelerator type specified in spec.labels.inference.optimization/acceleratorName to apply hardware-specific performance data.
  • The per-replica KV cache capacity, which is affected by dataLocal and the GPU memory available on each node.
  • The communication usage introduced by expert parallelism, which can affect the effective throughput per replica.

The following example shows a minimal spec.parallelism configuration for a Mixture of Experts (MoE) model:

Example MoE parallelism configuration

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: deepseek-r1
spec:
  model:
    uri: pvc://model-storage
    name: deepseek-ai/DeepSeek-R1-0528
  replicas: 1
  parallelism:
    data: 32
    dataLocal: 8
    expert: true
    tensor: 1
  # ...

spec.parallelism

The spec.parallelism fields control the following dimensions of distributed inference:

  • data: Specifies the total degree of data parallelism across the cluster. Combined with dataLocal, this value determines how many pod replicas are created. The replica count is derived from the formula data / dataLocal. For example, data: 32 with dataLocal: 8 produces 4 replicas.
  • dataLocal: Specifies the degree of data parallelism within a single pod. Each pod handles this many data-parallel shards locally. This value typically matches the number of GPUs per node.
  • expert: Enables expert parallelism for MoE models such as DeepSeek-R1 or Mixtral. When set to true, model experts are distributed across pods by using all-to-all communication backends. Set this to true only for MoE model architectures.
  • tensor: Specifies the degree of tensor parallelism within a pod. Controls how model layers are split across GPUs within a single pod. Set to 1 when using expert parallelism as the primary distribution strategy.
Replica count and resource requirements
The number of pods created by an LLMInferenceService deployment is determined by the parallelism configuration, not the replicas field alone. The effective replica count is calculated as data \ dataLocal. Each pod requires the number of GPUs specified by the dataLocal value. Ensure that your cluster has sufficient GPU capacity to support the total number of GPUs across all replicas.
Communication backends

The communication backend determines how pods exchange data during distributed inference. You configure the backend by setting the VLLM_ALL2ALL_BACKEND environment variable in the LLMInferenceService CR template, for example: 

  template:
    containers:
      - name: main
        env:
          # ...
          - name: VLLM_ALL2ALL_BACKEND
            value: deepep_high_throughput

The following backends are available:

  • deepep_high_throughput: Optimized for batch processing throughput. Requires RDMA-capable networking such as RoCE or InfiniBand. Use this backend for prefill-heavy workloads or high-throughput serving.
  • pplx: Optimized for low-latency decode operations. Requires RDMA-capable networking. Use this backend for latency-sensitive decode workloads.
  • naive: Uses TCP-based communication. Does not require RDMA hardware. Use this backend in environments without high-performance networking, such as cloud instances with standard virtual NICs.
Disaggregated serving and parallelism

When using prefill and decode separation, each stage can have its own parallelism settings. The prefill stage and decode stage run as independent deployment groups, each with its own replica count derived from the data \ dataLocal calculation.

In a disaggregated configuration, the WVA can scale each stage independently based on its workload characteristics. For example, the prefill stage might scale based on request queue depth while the decode stage scales based on KV cache utilization. KV cache transfer between stages is handled by the NixlConnector.

You can create an inference stack with an LLMInferenceService custom resource (CR). You can create multiple LLMInferenceService CRs within a single stack to serve the same model on different accelerator types.

There is a single inference stack per namespace, you cannot deploy multiple inference stacks in the same namespace. An inference stack consists of the following per-namespace resources:

  • HTTPRoute
  • InferencePool
  • Scheduler deployment that reconciles the InferencePool
  • Decode workload variant, deployed as a Deployment or LeaderWorkerSet
  • For prefill-decode disaggregation: one prefill workload variant, deployed as a Deployment or LeaderWorkerSet
  • For monitoring: ServiceMonitor resources for prefill and decode

The following resources are not part of the per-namespace inference stack:

  • Gateway: Either shared across multiple inference stacks or omitted in a gatewayless deployment
  • WVA controller: A single controller instance operates per cluster
  • OpenTelemetry Collector: Used only for distributed tracing and not part of the inference stack

You can enable intelligent autoscaling for your Distributed Inference with llm-d model deployments by configuring the workload variant autoscaler (WVA). The WVA controller is automatically deployed when the OpenShift AI Operator is installed. After you configure autoscaling in the LLMInferenceService custom resource, the WVA automatically adjusts the replica count of your model server based on real-time inference traffic and AI accelerator capacity.

Prerequisites

  • You have an OpenShift cluster on version 4.20 or later.
  • You have installed the OpenShift CLI (oc).
  • You have logged in as a user with cluster-admin privileges.

  • You have installed Red Hat OpenShift AI 3.5 with the Distributed Inference with llm-d stack enabled.

    Important

    Dependencies for installing and using the Distributed Inference with llm-d stack require configuration of the underlying Red Hat OpenShift AI deployment. You cannot use WVA without first configuring OpenShift AI.

  • Optional: You have installed jq.
  • You have installed compatible AI accelerators in the cluster.
  • You have enabled OpenShift User Workload Monitoring (UWM) in the cluster for Prometheus metrics collection. See Configuring user workload monitoring.
  • You have installed the custom-metrics-autoscaler Operator from OperatorHub. See Automatically scaling pods with the Custom Metrics Autoscaler Operator.
  • You have installed the Red Hat Connectivity Link Operator from OperatorHub. See Installing Red Hat Connectivity Link.

  • You have installed the Red Hat OpenShift Service Mesh 3 Operator in your cluster.

    Note

    The Operator is installed by default on any OpenShift cluster version 4.20 or later.

  • Optional: If you plan to use LeaderWorkerSet, you must install the Red Hat build of LeaderWorkerSet Operator in your cluster and create a LeaderWorkerSetOperator custom resource (CR). For more information, see LeaderWorkerSet Operator for Red Hat OpenShift.

  • You must have DataScienceClusterInitialization (DSCI) and DataScienceCluster (DSC) CRDs in your cluster, enabling the workload-variant-autoscaler-controller-manager, llmisvc-controller-manager and kserve-controller-manager controllers. The DataScienceClusterInitialization controller is automatically created by the OpenShift AI Operator. This example excerpt from the DataScienceCluster manifest enables the WVA controller: 

    apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: default-dsc
  labels:
    app.kubernetes.io/name: datasciencecluster
spec:
  components:
    kserve:
      # Create the KServe and LLMISVC controller managers
      managementState: "Managed"
      nim:
        managementState: "Managed"
      rawDeploymentServiceConfig: "Headed"
      wva:
        # Create the workload-variant-autoscaler controller manager
        managementState: "Managed"
        # ...Other DSC components as desired

  • You have confirmed that no other LLMInferenceService CR exists in the namespace you intend to deploy the WVA in.

    Important

    Each namespace should contain a single Distributed Inference with llm-d inference stack only.

Procedure

  1. Verify that the required controllers exist in the cluster. Run the following command: 

    $ oc get pods -n redhat-ods-applications \
  -l 'app.kubernetes.io/name in (kserve-controller-manager, llmisvc-controller-manager, workload-variant-autoscaler)'

    All controllers should report as ready: 

    NAME                                            READY   STATUS    RESTARTS   AGE
kserve-controller-manager                       1/1     Running   0          38m
llmisvc-controller-manager                      1/1     Running   0          38m
workload-variant-autoscaler-controller-manager  1/1     Running   0          38m

  2. Verify the default scaling configuration.

    Run the following command to inspect the workload-variant-autoscaler-saturation-scaling-config ConfigMap: 

    $ oc get cm workload-variant-autoscaler-saturation-scaling-config -n redhat-ods-applications -o jsonpath='{.data.default}'

    You should see the following default values: 

    kvCacheThreshold: 0.80
queueLengthThreshold: 5
kvSpareTrigger: 0.1
queueSpareTrigger: 3
enableLimiter: false

    These values represent the default scaling thresholds that the WVA uses to determine when to scale your inference workloads up or down.

  3. Grant the KEDA controller permissions to read from OpenShift monitoring resources.

    Note

    The Custom Metrics Autoscaler Operator does not automatically have permission to read metrics from the openshift-monitoring or openshift-user-workload-monitoring stack because OpenShift restricts access to cluster monitoring APIs. Access to Prometheus and Thanos endpoints requires explicit RBAC permissions such as the cluster-monitoring-view role.

    1. Authorize KEDA to read OpenShift user workload monitoring (UWM) metrics. Create the following CRs: 

      apiVersion: v1
kind: ServiceAccount
metadata:
  name: keda-metrics-reader
  namespace: openshift-keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-metrics-reader-monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-monitoring-view
subjects:
  - kind: ServiceAccount
    name: keda-metrics-reader
    namespace: openshift-keda
---
apiVersion: v1
kind: Secret
metadata:
  name: keda-metrics-reader-token
  namespace: openshift-keda
  annotations:
    kubernetes.io/service-account.name: keda-metrics-reader
type: kubernetes.io/service-account-token

    2. Allow KEDA to authenticate against OpenShift user workload monitoring. To give KEDA the proper permissions, create a ClusterTriggerAuthentication CR, which references the keda-metrics-reader-token and Secret CRs acting as a Token for the keda-metrics-reader and ServiceAccount CRs created in the previous step. Create the ClusterTriggerAuthentication CR: 

      apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: ai-inference-keda-thanos
spec:
  secretTargetRef:
    - parameter: bearerToken
      name: keda-metrics-reader-token
      key: token
    - parameter: ca
      name: keda-metrics-reader-token
      key: ca.crt

      KEDA can now pull metrics from the user workload monitoring stack.

  4. Patch the inferenceservice-config ConfigMap resource to use OpenShift User Workload Monitoring.

    By default, the inferenceservice-config ConfigMap has no autoscaling configuration set. Patch it to configure the llmisvc controller with the Thanos Querier URL to use for querying metrics, and the name of the ClusterTriggerAuthentication resource to use for authentication.

    1. Patch the ConfigMap resource: 

      $ oc patch configmap inferenceservice-config -n redhat-ods-applications \
    --type=json -p '[{"op":"replace","path":"/data/autoscaling-wva-controller-config","value":"{\"prometheus\":{\"url\":\"https://thanos-querier.openshift-monitoring.svc.cluster.local:9091\",\"authModes\":\"bearer\",\"triggerAuthName\":\"ai-inference-keda-thanos\",\"triggerAuthKind\":\"ClusterTriggerAuthentication\"}}"}]'

    2. Restart the llmisvc-controller-manager deployment to apply the configuration changes: 

      $ oc rollout restart deployment llmisvc-controller-manager -n redhat-ods-applications

      When the LLMISVC resource is created with autoscaling configurations, the llmisvc-controller-manager creates a KEDA ScaledObject with the settings embedded. This allows KEDA to connect to Thanos and authenticate its metric queries.

  5. Create an inference stack with autoscaling enabled:

    1. Create the namespace.

      Only one inference stack per namespace is supported. Create a dedicated namespace for the deployment. You can use any namespace, provided you adjust the manifests accordingly. 

      $ oc create ns autoscaling-example && oc project autoscaling-example

    2. Create the gateway.

      Create a gateway that uses TLS with OpenShift Service Mesh. OpenShift Service Mesh is installed by default on OpenShift version 4.20 and later. To create the Gateway, apply the following ConfigMap and Gateway manifests: 

      apiVersion: v1
kind: ConfigMap
metadata:
  name: autoscaling-example-gateway-config
  namespace: autoscaling-example
data:
  service: |
    metadata:
      annotations:
        service.beta.openshift.io/serving-cert-secret-name: "autoscaling-example-gateway-tls"
    spec:
      type: ClusterIP
  deployment: |
    spec:
      template:
        spec:
          containers:
            - name: istio-proxy
              resources:
                limits:
                  cpu: "16"
                  memory: 16Gi
                requests:
                  cpu: "4"
                  memory: 4Gi
      Note

      This example uses a ClusterIP service type for compatibility across OpenShift environments that do not have LoadBalancer service type integration. The istio-proxy resource limits are increased to handle the significant load generation required to trigger a scaling event during verification. Adjust these values according to your expected traffic patterns. 

      apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: autoscaling-example-gateway
  namespace: autoscaling-example
spec:
  gatewayClassName: data-science-gateway-class
  infrastructure:
    parametersRef:
      group: ""
      kind: ConfigMap
      name: autoscaling-example-gateway-config
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: autoscaling-example-gateway-tls
      mode: Terminate

    3. Create the LLMInferenceService CR with autoscaling configurations enabled. 

      apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: autoscaling-example-qwen
  namespace: autoscaling-example
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
    prometheus.io/path: "/metrics"
    security.opendatahub.io/enable-auth: 'false'
spec:
  router:
    scheduler: {}
    route: {}
    gateway:
      refs:
      - name: autoscaling-example-gateway
        namespace: autoscaling-example
  model:
    uri: hf://Qwen/Qwen2.5-7B-Instruct
    name: Qwen/Qwen2.5-7B-Instruct
  labels:
    inference.optimization/acceleratorName: H100
  scaling:
    minReplicas: 1
    maxReplicas: 5
    wva:
      keda:
        pollingInterval: 5
        cooldownPeriod: 30
  template:
    containers:
      - name: main
        image: registry.redhat.io/rhaii/vllm-cuda-rhel9:3.4.0
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: 1
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: 1
        startupProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS

      The following fields are important to understand when you customize the LLMInferenceService CR for your use case:

      • .metadata.annotations.security.opendatahub.io/enable-auth: Disables authentication and rate limiting for this example. In OpenShift AI, user authentication and rate limiting are handled by the Red Hat Connectivity Link Operator. This example disables authentication so that the load test can generate sufficient traffic to trigger a scaling event without being rate-limited. Production deployments should not set this annotation.
      • .spec.router.gateway.refs: Specifies the reference to the pre-created Gateway that the LLMInferenceService uses.
      • .spec.labels.inference.optimization/acceleratorName: Specifies the type of accelerator this workload uses, such as A100, H100, or cpu. The WVA uses this value to group variants by hardware type and look up accelerator-specific performance data when making scaling decisions. There is no fixed list of valid values: use whatever matches your hardware for semantic identification. If omitted, it defaults to unknown.
      • .spec.scaling: Specifies the user-facing scaling configurations at a per-LLMInferenceService level, which represents a per-inference-stack level. Do not confuse this with the configurations from the workload-variant-autoscaler-saturation-scaling-config ConfigMap, which represents the default scaling configurations that the workload-variant-autoscaler uses to determine the threshold at which it scales your inference workloads up or down.
      • .spec.template.containers[0].image: Specifies the inference image. Different inference images are used for different accelerators. If you are using accelerators, use the corresponding inference image.

      • .spec.template.containers[0].resources: Specifies the resource requests and limits. Adjust the resources as needed for more GPUs and add the tensor parallel flags to the VLLM_ADDITIONAL_ARGS environment variable, for example: 

        env:
  - name: VLLM_ADDITIONAL_ARGS
    value: "--tensor-parallel 2"
        Note

        The same requirement applies for accelerated networking devices like RoCE or InfiniBand. Autoscaling inter-node workloads through LeaderWorkerSet is not supported.

      • .spec.template.containers[0].startupProbe, .spec.template.containers[0].readinessProbe, .spec.template.containers[0].livenessProbe: Specifies the probe configurations. Because this example uses TLS end to end, ensure that your probes use the HTTPS scheme. If the probes do not use the TLS scheme, they fail to reach the endpoints and declare the pods as not ready or not healthy.

    4. Verify the LLMInferenceService and ScaledObject CRs were created successfully. 

      $ oc get llmisvc -n autoscaling-example
       
      NAME                        URL   READY   REASON   AGE
autoscaling-example-qwen         True             102s
       
      $ oc get scaledobject -n autoscaling-example
       
      NAME                                    SCALETARGETKIND      SCALETARGETNAME                    MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED   TRIGGERS     AUTHENTICATIONS            AGE
autoscaling-example-qwen-kserve-keda   apps/v1.Deployment   autoscaling-example-qwen-kserve   1     5     True    True     False      False    prometheus   ai-inference-keda-thanos   119s

Verification

Verify autoscaling behavior by confirming that inference server metrics appear in Prometheus, that the WVA emits scaling metrics, and that an end-to-end autoscaling event occurs.

  1. Verify that inference server component metrics appear in Prometheus.

    Ensure that the inputs to the WVA appear in Prometheus, specifically the inference server metrics: 

    $ TOKEN=$(oc whoami -t)
     
    $ THANOS=$(oc get route thanos-querier -n OpenShift-monitoring -o jsonpath='{.spec.host}')
for m in inference_extension_flow_control_queue_size inference_extension_flow_control_queue_size; do
  curl -sk -G -H "Authorization: Bearer $TOKEN" "https://$THANOS/api/v1/query" \
    --data-urlencode "query=${m}{exported_namespace=\"autoscaling-example\"}" \
    | jq '.data.result[0]'
done

    You should see three separate JSON payloads, one for each metric. The following example shows the expected output format: 

    {
  "metric": {
    "__name__": "vllm:num_requests_running",
    "container": "main",
    "endpoint": "8000",
    "engine": "0",
    "instance": "10.129.0.66:8000",
    "job": "autoscaling-example/kserve-llm-isvc-vllm-engine",
    "llm_isvc_component": "workload",
    "llm_isvc_name": "autoscaling-example-qwen",
    "llm_isvc_role": "both",
    "model_name": "Qwen/Qwen2.5-7B-Instruct",
    "namespace": "autoscaling-example",
    "pod": "autoscaling-example-qwen-kserve-649dc95869-ntlwq",
    "prometheus": "openshift-user-workload-monitoring/user-workload"
  },
  "value": [
    1774817937.592,
    "0"
  ]
}
    Note

    The WVA does not currently use metrics from the scheduler.

  2. Verify that the WVA is emitting metrics back to Prometheus.

    The metrics to look for are wva_current_replicas, wva_desired_replicas, and wva_desired_ratio. Run the following commands to query these metrics: 

    $ TOKEN=$(oc whoami -t)
     
    $ THANOS=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}')
for m in wva_desired_replicas wva_current_replicas wva_desired_ratio; do
  curl -sk -G -H "Authorization: Bearer $TOKEN" "https://$THANOS/api/v1/query" \
    --data-urlencode "query=${m}{exported_namespace=\"autoscaling-example\"}" \
    | jq '.data.result[0]'
done

    You should see JSON payloads for each metric. The following example shows the expected output format: 

    {
  "metric": {
    "__name__": "wva_desired_replicas",
    "accelerator_type": "H100",
    "endpoint": "https",
    "exported_namespace": "autoscaling-example",
    "instance": "10.128.0.64:8443",
    "job": "workload-variant-autoscaler-controller-manager-metrics-service",
    "namespace": "redhat-ods-applications",
    "pod": "workload-variant-autoscaler-controller-manager-85b8d895cd-x6gnr",
    "prometheus": "openshift-user-workload-monitoring/user-workload",
    "service": "workload-variant-autoscaler-controller-manager-metrics-service",
    "variant_name": "autoscaling-example-qwen-kserve-va"
  },
  "value": [
    1774819127.020,
    "1"
  ]
}

    At this point, verify the following resources that were created from your LLMInferenceService: 

    $ oc get va autoscaling-example-qwen-kserve-va -n autoscaling-example
     
    NAME                                  TARGET                             MODEL                      OPTIMIZED   METRICSREADY   AGE
autoscaling-example-qwen-kserve-va   autoscaling-example-qwen-kserve   Qwen/Qwen2.5-7B-Instruct   1           True           81m
     
    $ oc get scaledObject -n autoscaling-example
     
    NAME                                    SCALETARGETKIND      SCALETARGETNAME                    MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED   TRIGGERS     AUTHENTICATIONS            AGE
autoscaling-example-qwen-kserve-keda   apps/v1.Deployment   autoscaling-example-qwen-kserve   1     5     True    True     Unknown    False    prometheus   ai-inference-keda-thanos   83m

    The VariantAutoscaling CR should show METRICSREADY=True and the ScaledObject should show READY=true. Once the WVA controller starts reconciling the ScaledObject, it should also become ACTIVE=True. If you do not see these values, run oc describe on these resources and check their Status sections.

  3. Verify that the inference worker gets autoscaled.

    To make scaling events easier to trigger during testing, set the kvCacheThreshold key in the workload-variant-autoscaler-saturation-scaling-config ConfigMap in the redhat-ods-applications namespace to 0.10: 

    $ oc edit cm workload-variant-autoscaler-saturation-scaling-config -n redhat-ods-applications

    Change kvCacheThreshold from 0.80 to 0.10.

    The following script deploys a load-generator pod inside the cluster and watches for scaling events. The script resolves the gateway service, sends thousands of concurrent requests with long token generation to saturate the vLLM KV cache past the WVA scaling threshold, and monitors the replica count every 5 seconds:

    Note

    The scaling behavior and load values in this script are dependent on the accelerators in use. This example was tested with NVIDIA H100 AI accelerators. 

    #!/bin/bash
set -euo pipefail

NS="${1:-autoscaling-example}"
ISVC="${2:-autoscaling-example-qwen}"
CONCURRENCY="${3:-200}"
REQUESTS="${4:-5000}"
POD_NAME="load-test-$(date +%s)"
CM_NAME="script-${POD_NAME}"

GATEWAY_SVC=$(oc get svc -n "$NS" -l gateway.networking.k8s.io/gateway-name \
  -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "$GATEWAY_SVC" ]; then
  echo "ERROR: No gateway service found in namespace $NS"
  exit 1
fi
URL="https://${GATEWAY_SVC}.${NS}.svc.cluster.local/${NS}/${ISVC}/v1/chat/completions"

echo "=== WVA Autoscaling Test ==="
echo "URL:         $URL"
echo "Concurrency: $CONCURRENCY"
echo "Requests:    $REQUESTS"
echo ""
oc get deployment "${ISVC}-kserve" -n "$NS" \
  -o jsonpath='Initial state: {.spec.replicas}/{.status.readyReplicas} ready' && echo ""
echo ""

cleanup() {
  echo ""
  echo "Cleaning up..."
  oc delete pod "$POD_NAME" -n "$NS" --ignore-not-found --wait=false 2>/dev/null
  oc delete configmap "$CM_NAME" -n "$NS" --ignore-not-found 2>/dev/null
}
trap cleanup EXIT INT TERM

read -r -d '' LOAD_SCRIPT << 'EOF' || true
#!/bin/sh
set -e
URL="$1"; CONCURRENCY="$2"; REQUESTS="$3"

cat > /tmp/req.sh << 'REQEOF'
#!/bin/sh
curl -sk --max-time 600 "$1" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"Qwen/Qwen2.5-7B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Request $2. Write a detailed essay about topic $2 covering history, analysis, and predictions.\"}],\"max_tokens\":2048}" \
  -o /dev/null -w "req=$2 status=%{http_code} time=%{time_total}s\n"
REQEOF
chmod +x /tmp/req.sh

echo "Smoke test..."
STATUS=$(curl -sk --max-time 30 "$URL" \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Hi"}],"max_tokens":5}' \
  -o /dev/null -w "%{http_code}")
echo "Status: $STATUS"
if [ "$STATUS" != "200" ]; then
  echo "ERROR: Smoke test failed (HTTP $STATUS)"
  exit 1
fi

echo "Sending $REQUESTS requests ($CONCURRENCY concurrent)..."
seq 1 "$REQUESTS" | xargs -P "$CONCURRENCY" -I{} /tmp/req.sh "$URL" {}
echo "Done."
EOF

oc create configmap "$CM_NAME" -n "$NS" --from-literal=load.sh="$LOAD_SCRIPT"

cat <<MANIFEST | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: $POD_NAME
  namespace: $NS
spec:
  restartPolicy: Never
  containers:
  - name: load
    image: curlimages/curl
    command: ["sh", "/scripts/load.sh"]
    args: ["$URL", "$CONCURRENCY", "$REQUESTS"]
    resources:
      requests: { cpu: "4", memory: "4Gi" }
      limits:   { cpu: "8", memory: "8Gi" }
    volumeMounts:
    - { name: script, mountPath: /scripts }
  volumes:
  - name: script
    configMap: { name: $CM_NAME, defaultMode: 0755 }
MANIFEST

echo "Waiting for pod..."
oc wait --for=condition=Ready pod/"$POD_NAME" -n "$NS" --timeout=120s 2>/dev/null || true
sleep 2

POD_PHASE=$(oc get pod "$POD_NAME" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null)
if [ "$POD_PHASE" = "Failed" ]; then
  echo "ERROR: Pod failed:"
  oc logs "$POD_NAME" -n "$NS" 2>/dev/null
  exit 1
fi

oc logs -f "$POD_NAME" -n "$NS" 2>/dev/null &
LOGS_PID=$!

echo ""
echo "--- Watching replicas (Ctrl+C to stop) ---"
for _ in $(seq 1 120); do
  replicas=$(oc get deployment "${ISVC}-kserve" -n "$NS" -o jsonpath='{.spec.replicas}' 2>/dev/null)
  ready=$(oc get deployment "${ISVC}-kserve" -n "$NS" -o jsonpath='{.status.readyReplicas}' 2>/dev/null)
  echo "  [$(date +%H:%M:%S)] Replicas: ${replicas:-?} (${ready:-0} ready)"
  [ "${replicas:-1}" -gt 1 ] && echo "  ** Scale-up detected! **"
  oc get pod "$POD_NAME" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null \
    | grep -q "Succeeded\|Failed" && echo "  Load pod finished." && break
  sleep 5
done

kill "$LOGS_PID" 2>/dev/null || true
wait "$LOGS_PID" 2>/dev/null || true

    You should see the replica count increase as load is applied: 

      [17:30:03] Replicas: 1 (1 ready)
  [17:30:58] Replicas: 2 (1 ready)
  ** Scale-up detected! **

    During the load, you should see additional inference worker pods: 

    $ oc get pods -n autoscaling-example
     
    NAME                                                              READY   STATUS    RESTARTS   AGE
autoscaling-example-gateway-data-science-gateway-class-66c4qvpb   1/1     Running   0          81m
autoscaling-example-qwen-kserve-56b64d69d-b297x                  1/1     Running   0          106m
autoscaling-example-qwen-kserve-56b64d69d-kvvmc                  1/1     Running   0          10m
autoscaling-example-qwen-kserve-56b64d69d-m78nz                  1/1     Running   0          7m1s
autoscaling-example-qwen-kserve-router-scheduler-77cd5549p4w95   2/2     Running   0          106m

    After the load ceases, the deployment scales back down to 1 replica: 

    NAME                                                              READY   STATUS    RESTARTS   AGE
autoscaling-example-gateway-data-science-gateway-class-66c4qvpb   1/1     Running   0          81m
autoscaling-example-qwen-kserve-56b64d69d-b297x                  1/1     Running   0          106m
autoscaling-example-qwen-kserve-router-scheduler-77cd5549p4w95   2/2     Running   0          106m
    Note

    The WVA uses a stair-step methodology to scaling, imposing a stabilization window on scaling events so that a traffic burst does not greedily attempt to occupy all your GPUs. If load continues past the stabilization window, you may see additional scaling events.

9.6. VariantAutoscaling CR configuration reference
Copy link

You configure autoscaling behavior by creating a VariantAutoscaling custom resource (CR).

Important

When multiple configuration sources are available, the workload variant autoscaler applies the following precedence order, from highest to lowest.

  1. CLI flags
  2. Environment variables
  3. ConfigMap values
  4. Built-in defaults

Namespace-scoped ConfigMap resources override cluster-level ConfigMap resources for the same parameters. To enable namespace-scoping, add the wva.llmd.ai/config-enabled: "true" label to the target namespace.

The following is an example VariantAutoscaling custom resource (CR). 

apiVersion: llmd.ai/v1alpha1
kind: VariantAutoscaling
metadata:
  name: llama-70b-premium-h100
  namespace: llm-inference
  labels:
    app: llm-inference
    model: llama-70b
    variant: premium
    accelerator: h100
spec:
  scaleTargetRef:
    kind: Deployment
    name: llama-70b-premium-h100
  modelID: "meta/llama-3.1-70b"
  minReplicas: 1
  maxReplicas: 5
  • metadata.name: Specifies a name for the VariantAutoscaling resource. Use a descriptive name that identifies the model, variant role, and accelerator type.
  • metadata.namespace: Specifies the namespace where the target deployment runs. Each namespace supports only one Distributed Inference with llm-d inference stack.
  • metadata.labels: Specifies optional labels for operational tracking. The workload variant autoscaler (WVA) does not use these labels for scaling decisions, but they are useful for filtering and monitoring. The accelerator type is determined by the deployment’s actual hardware, not by these labels.
  • spec.scaleTargetRef.kind: Required. Specifies the kind of the Kubernetes resource that the WVA scales. Supported value: Deployment.
  • spec.scaleTargetRef.name: Specifies the name of the Kubernetes resource that the WVA scales. Must match the name of an existing resource in the same namespace.
  • spec.modelID: Specifies the OpenAI API-compatible model identifier for the inference workload.
  • spec.minReplicas: Specifies the minimum number of replicas. The WVA does not scale below this value. Must be 1 or greater. Default: 1.
  • spec.maxReplicas: Specifies the maximum number of replicas. The WVA does not scale above this value. Set this value based on your cluster AI accelerator capacity. Default: 2.

The following reference LLMInferenceService custom resource (CR) describes a solution for deploying and autoscaling a DeepSeek-Coder-V2-Lite MoE model with expert parallelism, LeaderWorkerSet, and the workload variant autoscaler (WVA) across multiple NVIDIA GPU nodes on Google Cloud Platform (GCP).

Note

The NVIDIA Collective Communications Library (NCCL) environment variables in this configuration are tuned for environments that use TCP-based networking and do not have Remote Direct Memory Access (RDMA) available. If your cluster nodes use RDMA-capable networking such as InfiniBand or RoCE, you must adjust the NCCL settings accordingly.

Reference DeepSeek-Coder-V2-Lite model deployment with workload variant autoscaler on GCP

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: deepseek-coder-v2
  namespace: autoscaling-example
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
    prometheus.io/path: "/metrics"
    security.opendatahub.io/enable-auth: 'false'
spec:
  model:
    # Alternatively, use pvc:// URI for pre-initialized PVC storage for the model
    uri: hf://deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
    name: deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
  labels:
    inference.optimization/acceleratorName: H100 # Match AI accelerator profile name
  parallelism:
    data: 2
    dataLocal: 1
    expert: true
    tensor: 1
  router:
    scheduler: { }
    route: { }
    gateway:
      refs:
      - name: autoscaling-example-gateway
        namespace: autoscaling-example
  scaling:
    minReplicas: 1
    maxReplicas: 5
    wva:
      keda:
        pollingInterval: 5
        cooldownPeriod: 30
  template:
    containers:
      - name: main
        env:
          - name: VLLM_API_SERVER_COUNT
            value: "1"
          - name: VLLM_LOGGING_LEVEL
            value: INFO
          # GCP gVNIC: Optimized for TCP-based high-performance networking
          # No RDMA devices available, using gVNIC with NCCL TCP optimization
          - name: CUDA_DEVICE_ORDER
            value: "PCI_BUS_ID"
          # vLLM memory and performance settings
          - name: VLLM_ADDITIONAL_ARGS
            value: "--gpu-memory-utilization 0.95 --max-model-len 8192 --enforce-eager"
          - name: VLLM_ALL2ALL_BACKEND
            value: naive
          - name: PYTORCH_CUDA_ALLOC_CONF
            value: "expandable_segments:True"
          # NCCL settings for TCP over gVNIC
          - name: NCCL_IB_DISABLE
            value: "1"  # No InfiniBand available
          - name: NCCL_NET_GDR_LEVEL
            value: "0"  # No GPUDirect RDMA
          - name: NCCL_P2P_LEVEL
            value: "NVL"  # NVLink for intra-node P2P
          - name: NCCL_SOCKET_IFNAME
            value: eth0
          - name: NCCL_NSOCKS_PERTHREAD
            value: "2"  # Lower count reduces memory pressure
          - name: NCCL_SOCKET_NTHREADS
            value: "2"
          - name: NCCL_BUFFSIZE
            value: "2097152"  # 2 MB
          - name: NCCL_DEBUG
            value: WARN
          # NVSHMEM over UCX/TCP
          - name: NVSHMEM_REMOTE_TRANSPORT
            value: "ucx"
          - name: NVSHMEM_DISABLE_CUDA_VMM
            value: "0"
          - name: NVSHMEM_BOOTSTRAP_TWO_STAGE
            value: "1"
          - name: NVSHMEM_BOOTSTRAP_TIMEOUT
            value: "300"
          - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
            value: eth0
          - name: NVSHMEM_DEBUG
            value: INFO
          # UCX transport layers
          - name: UCX_TLS
            value: "tcp,sm,self,cuda_copy,cuda_ipc"
          - name: UCX_NET_DEVICES
            value: eth0
          # Intra-node GPU-to-GPU transfers
          - name: NVIDIA_GDRCOPY
            value: enabled
        resources:
          limits:
            cpu: 128
            ephemeral-storage: 100Gi
            memory: 512Gi
            nvidia.com/gpu: "2"
            # No rdma/roce_gdr needed for TCP over gVNIC
          requests:
            cpu: 64
            ephemeral-storage: 100Gi
            memory: 256Gi
            nvidia.com/gpu: "2"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 4800
          periodSeconds: 10
          timeoutSeconds: 10
          failureThreshold: 3
  worker:
    containers:
      - name: main
        env:
          - name: VLLM_API_SERVER_COUNT
            value: "1"
          - name: VLLM_LOGGING_LEVEL
            value: INFO
          - name: CUDA_DEVICE_ORDER
            value: "PCI_BUS_ID"
          # vLLM memory and performance settings
          - name: VLLM_ADDITIONAL_ARGS
            value: "--gpu-memory-utilization 0.95 --max-model-len 8192 --enforce-eager"
          - name: VLLM_ALL2ALL_BACKEND
            value: naive
          - name: PYTORCH_CUDA_ALLOC_CONF
            value: "expandable_segments:True"
          # NCCL settings for TCP over gVNIC
          - name: NCCL_IB_DISABLE
            value: "1"  # No InfiniBand available
          - name: NCCL_NET_GDR_LEVEL
            value: "0"  # No GPUDirect RDMA
          - name: NCCL_P2P_LEVEL
            value: "NVL"  # NVLink for intra-node P2P
          - name: NCCL_SOCKET_IFNAME
            value: eth0
          - name: NCCL_NSOCKS_PERTHREAD
            value: "2"  # Reduce sockets to avoid allocation errors
          - name: NCCL_SOCKET_NTHREADS
            value: "2"
          - name: NCCL_BUFFSIZE
            value: "2097152"  # 2MB buffer to reduce memory usage
          - name: NCCL_DEBUG
            value: WARN
          # NVSHMEM over UCX/TCP
          - name: NVSHMEM_REMOTE_TRANSPORT
            value: "ucx"
          - name: NVSHMEM_DISABLE_CUDA_VMM
            value: "0"
          - name: NVSHMEM_BOOTSTRAP_TWO_STAGE
            value: "1"
          - name: NVSHMEM_BOOTSTRAP_TIMEOUT
            value: "300"
          - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
            value: eth0
          - name: NVSHMEM_DEBUG
            value: INFO
          # UCX transport layers
          - name: UCX_TLS
            value: "tcp,sm,self,cuda_copy,cuda_ipc"
          - name: UCX_NET_DEVICES
            value: eth0
          # Intra-node GPU-to-GPU transfers
          - name: NVIDIA_GDRCOPY
            value: enabled
        resources:
          limits:
            cpu: 128
            ephemeral-storage: 100Gi
            memory: 512Gi
            nvidia.com/gpu: "2"
            # No rdma/roce_gdr needed for TCP over gVNIC
          requests:
            cpu: 64
            ephemeral-storage: 100Gi
            memory: 256Gi
            nvidia.com/gpu: "2"

9.8. Saturation scaling ConfigMap reference
Copy link

The workload variant autoscaler (WVA) uses a ConfigMap named workload-variant-autoscaler-saturation-scaling-config for saturation-based scaling thresholds.

Each ConfigMap contains two types of entries:

  • A default entry that contains global parameters that apply to all models in the scope (cluster-wide or namespace-scoped)
  • Optional override entries that contain model-specific customizations with arbitrary names, such as granite-override or llama-override

The following example shows a ConfigMap CR with cluster-level defaults and a model-specific override:

The default entry contains global parameters that apply to all models unless a matching override exists. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: workload-variant-autoscaler-saturation-scaling-config
  namespace: redhat-ods-applications
data:
  default: |
    kvCacheThreshold: <kv_cache_threshold>
    queueLengthThreshold: <queue_length_threshold>
    kvSpareTrigger: <kv_spare_trigger>
    queueSpareTrigger: <queue_spare_trigger>
    enableLimiter: <enable_limiter>
  <override_name>: |
    model_id: "<model_id>"
    namespace: "<namespace>"
    kvCacheThreshold: <kv_cache_threshold>
    kvSpareTrigger: <kv_spare_trigger>
  • kvCacheThreshold: Specifies the KV cache utilization threshold as a percentage. A replica is considered saturated if its KV cache utilization is greater than or equal to this value. Valid range: 0.0 to 1.0. Lower values trigger scale-up sooner. Default: 0.80.
  • queueLengthThreshold: Specifies the queue length threshold. A replica is considered saturated if its queue length is greater than or equal to this value. Lower values trigger scale-up sooner. Default: 5.
  • kvSpareTrigger: Specifies the KV cache spare capacity trigger. The WVA signals a scale-up if the average spare KV capacity across non-saturated replicas falls below this value. Valid range: 0.0 to 1.0. Default: 0.10.
  • queueSpareTrigger: Specifies the queue spare capacity trigger. The WVA signals a scale-up if the average spare queue capacity across non-saturated replicas falls below this value. Default: 3.
  • enableLimiter: Enables the GPU limiter to constrain scaling based on available cluster GPU capacity. When true, the WVA limits desired replicas so that deployments do not request more GPUs than are available in the cluster, preventing new replicas from entering a pending state. Default: false.
  • <override_name>: Override entry name, such as llm-d or granite-override. The WVA matches overrides by using the model_id and namespace fields, not the entry name.
  • model_id: Specifies the model identifier that this override applies to, such as Qwen/Qwen3-0.6B or ibm/granite-13b. The WVA uses this value together with namespace to match the override to a specific model deployment.
  • namespace: Specifies the namespace of the model deployment that this override applies to. The WVA uses this value together with model_id to match the override to a specific model deployment.
Note

The WVA resolves configuration by first looking for an override that matches both model_id and namespace. If no matching override is found, the WVA uses the default entry. Any parameters that are not specified in the matched entry receive hardcoded default values.

9.9. Autoscaling metrics
Copy link

The workload variant autoscaler (WVA) exposes the following autoscaling-specific metrics through Prometheus. These metrics reflect the scaling decisions and state of the WVA controller.

Expand
Table 9.1. WVA autoscaling metrics
Metric nameTypeLabelsDescription

wva_current_replicas

gauge

variant_name, namespace, accelerator_type

The current number of replicas managed by the WVA for each VariantAutoscaling resource.

wva_desired_replicas

gauge

variant_name, namespace, accelerator_type

The target number of replicas calculated by the WVA. This is the metric read by KEDA or HPA for actuation.

wva_replica_scaling_total

counter

variant_name, namespace, direction, reason

The total number of replica scaling events triggered by the WVA. The direction label is up or down. The reason label indicates the cause of the scaling event.

wva_desired_ratio

gauge

variant_name, namespace, accelerator_type

The ratio of desired replicas to current replicas, indicating the scaling pressure on the deployment.

The WVA uses the following vLLM inference metrics as input signals for scaling decisions. These metrics are exposed by the vLLM model server, not by the WVA controller.

Expand
Table 9.2. vLLM saturation metrics used by the WVA
Metric nameDescription

vllm:kv_cache_usage_perc

KV cache utilization as a percentage. A value of 0.0 to 1.0. The WVA uses max_over_time over a 1-minute window to determine saturation.

vllm:num_requests_waiting

Number of requests waiting to be processed. The WVA uses max_over_time over a 1-minute window to determine queue depth.

Note

The vLLM saturation metrics must include pod, model_name, and namespace labels for the WVA to correlate metrics with the correct VariantAutoscaling resource.

9.10. HPA behavior defaults
Copy link

The workload variant autoscaler (WVA) uses KEDA to manage HPA resources. KEDA configures the following default HPA scaling behavior. These values control the stabilization window and scaling velocity.

Expand
Table 9.3. HPA scaling behavior defaults
ParameterDefaultDescription

scaleUp.stabilizationWindowSeconds

240

The stabilization window for scale-up decisions, in seconds. The HPA waits this long after the last scaling event before scaling up again.

scaleDown.stabilizationWindowSeconds

240

The stabilization window for scale-down decisions, in seconds.

selectPolicy

Max

The policy selection strategy. Max selects the policy that results in the largest change.

policies[0].type

Pods

The scaling policy type.

policies[0].value

10

The maximum number of pods to add or remove per scaling period.

policies[0].periodSeconds

150

The duration of the scaling period, in seconds.

If your workload variant autoscaler (WVA) deployment is not scaling as expected, use the following troubleshooting steps to diagnose and resolve common issues.

  • Check the WVA controller status:

    View WVA controller logs to diagnose issues: 

    $ oc logs -l app.kubernetes.io/name=workload-variant-autoscaler -n redhat-ods-applications

  • Verify VariantAutoscaling resource conditions:

    Check the status conditions on your VariantAutoscaling resources: 

    $ oc describe va <variant_name> -n <namespace>

    Common condition states:

    • MetricsAvailable: True with reason MetricsFound: Prometheus metrics are available
    • MetricsAvailable: False with reason MetricsMissing: No metrics found for the variant. Check that vLLM metrics are being scraped.
    • MetricsAvailable: False with reason PrometheusError: WVA cannot connect to Prometheus. Check Prometheus configuration and network connectivity.
    • OptimizationReady: True with reason OptimizationSucceeded: WVA successfully calculated scaling decisions
    • OptimizationReady: False with reason OptimizationFailed: Scaling calculation failed. Check WVA logs for details.

  • Verify that KEDA resources are functioning correctly:

    Check the status of the KEDA ScaledObject resources in your namespace: 

    $ oc get scaledobjects -n <namespace>

    Check the KEDA operator logs for errors: 

    $ oc logs -l app=keda-operator -n openshift-keda

  • Verify Prometheus metrics are being collected:

    Confirm that vLLM metrics are available in Prometheus: 

    $ oc port-forward -n openshift-monitoring prometheus-k8s-0 9090:9090

    Then query Prometheus at http://localhost:9090 for vllm:kv_cache_usage_perc and vllm:num_requests_waiting.

  • Check the WVA scaling metrics in Prometheus:

    Query the wva_desired_replicas metric to verify the WVA is calculating scaling targets: 

    $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'wva_desired_replicas'

  • Enable user workload monitoring:

    If metrics are not being collected, verify that OpenShift User Workload Monitoring is enabled: 

    $ oc get configmap cluster-monitoring-config -n openshift-monitoring

    If the ConfigMap does not exist, create it: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top