Chapter 6. Validate inference workload changes with controlled deployment

Procedure

1. Add the group and weight fields to your existing production LLMInferenceService if they are not already set.

   Open your production LLMInferenceService manifest and add spec.router.route.group and spec.router.route.weight:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: <production-llmisvc-name>
     namespace: <namespace>
   spec:
     model:
       name: <model-name>
       uri: <model-uri>
     router:
       route:
         group: <group-name>
         weight: 9
       gateway: {}
       scheduler: {}
     # ...existing spec fields...

   where:

   _<production-llmisvc-name>_

   Specifies the name of your existing production LLMInferenceService.

   _<namespace>_

   Specifies the namespace where the production deployment runs.

   _<model-name>_

   Specifies the model name.

   _<model-uri>_

   Specifies the URI from where the model is downloaded.

   _<group-name>_

   Specifies the routing group name. Use 1 to 63 lowercase alphanumeric characters or hyphens, matching the pattern ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$.

   Apply the updated manifest:

   $ oc apply -f <production-llmisvc-file>.yaml

2. Create a manifest for the canary LLMInferenceService with the same routing group name as the production version, starting at weight 1:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: <canary-llmisvc-name>
     namespace: <namespace>
   spec:
     model:
       name: <model-name>
       uri: <canary-model-uri>
     replicas: <replica-count>
     router:
       route:
         group: <group-name>
         weight: 1
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         resources:
           limits:
             nvidia.com/gpu: "<gpu-count>"
           # ...resource configuration...

   where:

   _<canary-llmisvc-name>_

   Specifies a name for the canary version that is distinct from the production version name.

   _<canary-model-uri>_

   Specifies the model URI for the canary version, such as a new model revision or engine build.

   _<group-name>_

   Specifies the same routing group name used by the production version.

   _<model-name>_

   Specifies the model name for the canary version. In a typical canary deployment, use the same model name as the production version.

   Important

   Start the canary weight at a small nonzero value such as 1 rather than 0. Deploying at weight 0 and then increasing the weight later can cause transient connection failures while the backend warms up. A small nonzero weight ensures the backend cluster is warmed immediately with minimal traffic. With a production weight of 9 and a canary weight of 1, approximately 10% of traffic goes to the canary version.

3. Apply the canary manifest:

   $ oc apply -f <canary-llmisvc-file>.yaml

4. Wait for the GroupReady condition to become True:

   $ oc wait llminferenceservice <canary-llmisvc-name> -n <namespace> \
       --for=condition=GroupReady --timeout=300s

5. Verify the routing group membership by listing all LLMInferenceService resources in the namespace:

   $ oc get llminferenceservice -n <namespace> \
       -o custom-columns=NAME:.metadata.name,GROUP:.status.router.group.name,WEIGHT:.spec.router.route.weight,READY:.status.conditions[?(@.type=="GroupReady")].status

   The output lists all group members with their configured weights and GroupReady status:

   NAME                            GROUP            WEIGHT   READY
   _<production-llmisvc-name>_     _<group-name>_   9        True
   _<canary-llmisvc-name>_         _<group-name>_   1        True

   Verify that both versions belong to the same routing group and that GroupReady is True on both.

6. Adjust the traffic split to shift more traffic to the canary by updating the weight fields on both versions.

   For example, to shift to a 50/50 split:

   $ oc patch llminferenceservice <production-llmisvc-name> -n <namespace> \
       --type merge -p {"spec":{"router":{"route":{"weight":5}}}}
   $ oc patch llminferenceservice <canary-llmisvc-name> -n <namespace> \
       --type merge -p {"spec":{"router":{"route":{"weight":5}}}}

Procedure

1. Create a PodMonitor to scrape vLLM metrics with per-version labeling.

   vLLM exposes metrics on port 8000, but its model_name label is the same across versions that serve the same model. To distinguish versions, the PodMonitor copies the pod’s app.kubernetes.io/name label into an llm_isvc_name metric label:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: llmisvc-vllm
     namespace: <namespace>
   spec:
     selector:
       matchLabels:
         kserve.io/component: workload
     podMetricsEndpoints:
       - portNumber: 8000
         path: /metrics
         interval: 10s
         params:
           format:
             - prometheus
         relabelings:
           - sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
             targetLabel: llm_isvc_name

   Apply the PodMonitor:

   $ oc apply -f <podmonitor-file>.yaml

2. Verify that per-version metrics are available in Prometheus by checking for distinct llm_isvc_name values:

   vllm:request_success_total{llm_isvc_name=~"<v1-name>|<v2-name>"}

   Replace _<v1-name>_ and _<v2-name>_ with the LLMInferenceService names for each version. If the llm_isvc_name label is missing, verify that the PodMonitor is active and that the relabeling rule is correct.

3. Verify that the actual traffic distribution matches your configured weights by comparing request rates across versions:

   sum by(llm_isvc_name) (
     rate(vllm:request_success_total{llm_isvc_name=~"<v1-name>|<v2-name>"}[5m])
   )

   The ratio of request rates between versions should approximate the ratio of configured weights. For example, a 9:1 weight split should produce approximately 90% and 10% of total requests per version.

4. Compare time to first token across versions by running the following PromQL query in the OpenShift web console or your Prometheus interface:

   histogram_quantile(0.95,
     sum by(le, llm_isvc_name) (
       rate(vllm:time_to_first_token_seconds_bucket{llm_isvc_name=~"<v1-name>|<v2-name>"}[5m])
     )
   )

5. Compare additional key metrics using PromQL queries filtered by llm_isvc_name.

   The following table lists recommended metrics for promotion decisions.

   Table 6.1. Key metrics for cross-version comparison

   Metric	PromQL query pattern
   Generation throughput	sum by(llm_isvc_name) (rate(vllm:generation_tokens_total{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m]))
   Request success rate	sum by(llm_isvc_name) (rate(vllm:request_success_total{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m]))
   Queue depth	sum by(llm_isvc_name) (vllm:num_requests_waiting{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"})
   End-to-end latency (P95)	histogram_quantile(0.95, sum by(le, llm_isvc_name) (rate(vllm:e2e_request_latency_seconds_bucket{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m])))
   KV cache utilization	avg by(llm_isvc_name) (vllm:kv_cache_usage_perc{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"})

6. If end-to-end latency is higher than expected, isolate whether the cause is a capacity issue or an engine bottleneck by comparing queue time against prefill and decode time:

   Table 6.2. Latency breakdown queries

   Component	PromQL query pattern
   Queue time (P95)	histogram_quantile(0.95, sum by(le, llm_isvc_name) (rate(vllm:request_queue_time_seconds_bucket{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m])))
   Prefill time (P95)	histogram_quantile(0.95, sum by(le, llm_isvc_name) (rate(vllm:request_prefill_time_seconds_bucket{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m])))
   Decode time (P95)	histogram_quantile(0.95, sum by(le, llm_isvc_name) (rate(vllm:request_decode_time_seconds_bucket{llm_isvc_name=~"_<v1-name>_|_<v2-name>_"}[5m])))

   If queue time is significantly higher than prefill and decode time combined, the version needs more replicas. If prefill or decode time is high but queue time is low, the latency is an engine-level bottleneck rather than a capacity issue.

7. Based on the metric comparison, decide whether to promote or roll back.

   Table 6.3. Decision criteria for controlled deployment

   Signal	Promote	Roll back
   Error rate	Equal to or lower than baseline	Higher than baseline
   Time to first token	Within acceptable threshold of baseline	Significantly higher than baseline
   End-to-end latency	Within acceptable threshold of baseline	Significantly higher than baseline
   Queue depth	Stable and not growing	Growing, indicating the version cannot handle its traffic share
   KV cache utilization	Within expected bounds	Approaching capacity or significantly higher than baseline
   Pod health	All pods running and ready	Pod restarts, readiness failures, or reduced pod count

Procedure

1. To promote the canary version, set the production version weight to 0 so that all traffic goes to the canary:

   $ oc patch llminferenceservice <production-llmisvc-name> -n <namespace> \
       --type merge -p {"spec":{"router":{"route":{"weight":0}}}}

   The production version pods remain active but idle. Because the production version still exists in the routing group at weight 0, you can roll back by restoring its weight.

2. To roll back during a controlled deployment, restore the original version weight and set the canary version weight to 0:

   $ oc patch llminferenceservice <production-llmisvc-name> -n <namespace> \
       --type merge -p {"spec":{"router":{"route":{"weight":9}}}}
   $ oc patch llminferenceservice <canary-llmisvc-name> -n <namespace> \
       --type merge -p {"spec":{"router":{"route":{"weight":0}}}}

   Note

   Rollback options depend on the state of the original version:

   • If the original version still exists with weight 0, adjust weights back to restore traffic.
   • If the original version was stopped with the serving.kserve.io/stop annotation, remove the annotation and restore the weight.
   • If the original version was deleted, you must redeploy it from the original manifest.

3. Optional: To reclaim GPU resources from a version that is no longer receiving traffic, force-stop it by applying the serving.kserve.io/stop annotation:

   $ oc annotate llminferenceservice <llmisvc-name> -n <namespace> \
       serving.kserve.io/stop=true

   A stopped member remains in the routing group but is omitted from backendRef entries in peer members' HTTPRoutes, regardless of the spec.router.route.weight value. The member stopped field in status.router.group.members is set to true.

   To resume a stopped version, remove the annotation:

   $ oc annotate llminferenceservice <llmisvc-name> -n <namespace> \
       serving.kserve.io/stop-

4. To delete an old version after successful promotion, delete the LLMInferenceService resource:

   $ oc delete llminferenceservice <old-llmisvc-name> -n <namespace>

   Warning

   Do not delete a version that has a weight greater than 0. Deleting a member with active traffic causes 500 errors for in-flight requests until the remaining group members reconcile. Always set the weight to 0 and verify that the GroupReady condition reflects the updated group membership before deleting.

   The control plane holds the deletion until peer members remove the corresponding backendRef from their HTTPRoutes, so in-flight requests to other members are not disrupted.