Chapter 8. Managing mixed workloads with priority queuing

1. The Gateway’s AuthPolicy automatically injects two headers based on authentication:

   • x-gateway-inference-fairness-id - Groups requests from the same authentication source for fair queuing
   • x-gateway-inference-objective - Determines request priority by matching an InferenceObjective resource
2. The request enters a priority-aware queue in the Endpoint Picker and waits for dispatch.

3. The dispatch logic applies strict priority ordering:

   • Higher-priority requests dispatch before lower-priority requests
   • Requests at the same priority level use fairness policies. The default global-strict-fairness-policy is a greedy strategy that ignores tenant boundaries and offers no isolation. A single tenant in a priority can starve other tenants of the same priority. For equitable service distribution, configure round-robin-fairness-policy

4. A saturation check monitors queue depth and KV cache utilization on model servers:

   • When pool saturation is detected (saturation >= 1), dispatch is halted for all priority bands
   • All requests wait in their respective priority queues until capacity becomes available
5. When capacity is available, the request proceeds to the Scheduling layer and routes to a specific model server.

Procedure

1. Configure the Endpoint Picker to enable flow control and configure saturation detection. The EndpointPickerConfig is embedded within the LLMInferenceService custom resource under spec.router.scheduler. Edit your LLMInferenceService resource:

   $ oc edit llminferenceservice <llm-service-name> -n <namespace>

2. Locate the spec.router.scheduler section and add the flowControl feature gate and configure the saturationDetector:

   apiVersion: inference.networking.x-k8s.io/v1alpha1
   kind: EndpointPickerConfig
   featureGates:
     - "flowControl"
   saturationDetector:
     queueDepthThreshold: 5
     kvCacheUtilThreshold: 0.8
     metricsStalenessThreshold: 200ms

   where:

   • queueDepthThreshold: Specifies the queue depth threshold on model servers. A value of 5 (default) balances latency and throughput. Set to 1 for maximum fairness (forces all queuing into EPP priority-aware queues). Set to 1-2x max batch size for maximum throughput (allows model servers to form fuller batches). To decide your model server’s batch size for tuning queueDepthThreshold, check the vLLM deployment’s --max-batch-total-tokens parameter or refer to your LLMInferenceService configuration. If unsure, start with the default value of 5 and adjust based on observed latency and throughput metrics.
   • kvCacheUtilThreshold: Specifies the KV cache utilization threshold (0.0 to 1.0). Default is 0.8.
   • metricsStalenessThreshold: Specifies how long metrics can be stale before being considered invalid. Default is 200ms.
3. Save and exit the editor.

4. Create an InferenceObjective resource to define a priority tier:

   apiVersion: inference.networking.x-k8s.io/v1alpha2
   kind: InferenceObjective
   metadata:
     name: <objective-name>
     namespace: <namespace>
   spec:
     priority: <priority-value>
     poolRef:
       group: inference.networking.k8s.io
       kind: InferencePool
       name: <inference-pool-name>

   where:

   • metadata.name - Specifies the InferenceObjective name.
   • metadata.namespace - Specifies the namespace where the InferencePool is deployed.

   • priority - Specifies the priority value. Higher integer values represent higher priority. Negative values designate lower-priority requests that are queued and dispatched last. Requests can be rejected when:

     • The priority band’s capacity is exhausted (HTTP 429)
     • They time out (HTTP 503)
     • The client disconnects abruptly (HTTP 503)
   • poolRef.name - Specifies the name of the InferencePool resource this objective applies to.

5. Apply the InferenceObjective resource:

   $ oc apply -f <inference-objective-file>.yaml

6. Optional: Configure advanced flow control settings for priority bands.

   After creating InferenceObjective resources, you can customize flow control behavior by configuring priority band capacity, request timeouts, and fairness policies. Edit your LLMInferenceService resource and add the flowControl configuration under spec.router.scheduler:

   apiVersion: inference.networking.x-k8s.io/v1alpha1
   kind: EndpointPickerConfig
   featureGates:
     - "flowControl"
   flowControl:
     maxBytes: 1073741824
     defaultRequestTTL: 1m
     priorityBands:
     - priority: 1
       orderingPolicyRef: fcfs-ordering-policy
       fairnessPolicyRef: round-robin-fairness-policy
     - priority: 0
       orderingPolicyRef: fcfs-ordering-policy
       fairnessPolicyRef: round-robin-fairness-policy
     - priority: -1
       orderingPolicyRef: fcfs-ordering-policy
       fairnessPolicyRef: round-robin-fairness-policy

   where:

   • maxBytes - Specifies the priority band’s capacity in bytes. When this limit is exceeded, requests receive HTTP 429 errors. Default is 1073741824 (1 GiB).
   • defaultRequestTTL - Specifies the request timeout. Requests waiting longer than this duration receive HTTP 503 errors. Default is 1 minute.
   • priorityBands[].priority - Specifies the priority value for this band. Must match the priority values defined in your InferenceObjective resources.
   • priorityBands[].orderingPolicyRef - Specifies the ordering policy. Default is fcfs-ordering-policy (first-come, first-served).
   • priorityBands[].fairnessPolicyRef - Specifies the fairness policy. The default global-strict-fairness-policy offers no tenant isolation and can result in starvation. For equitable service distribution across tenants, use round-robin-fairness-policy.

Verification

1. Send sustained traffic with varying request priorities using load testing tools.

   Priority differentiation is active whenever there are multiple requests waiting in the queue. The saturation signal controls whether anything gets dispatched at all. Use load testing tools to send sustained traffic with varying request priorities, tenant fairness IDs, and concurrent request volumes.

2. Monitor flow control metrics in Prometheus:

   • inference_extension_flow_control_queue_size - Queue depth for requests waiting to dispatch
   • inference_extension_flow_control_request_queue_duration_seconds - Time requests sit in the flow control queue
3. If queue depth remains at 0, increase concurrency to create saturation conditions where priority differentiation occurs.