Red Hat SummitChapter 9. Distributed Inference with llm-d Observability
You can use Observability to analyze your llm-d deployments, improving overall model management and monitoring on the platform.
Use Prometheus to collect metrics from Distributed Inference with llm-d deployments and build custom observability dashboards to monitor token throughput, cache utilization, routing decisions, and inference performance. On OpenShift, the controller automatically creates PodMonitor and ServiceMonitor resources to scrape metrics from vLLM engines and the inference scheduler. On other Kubernetes platforms, configure Prometheus scrape targets manually to collect metrics from the vLLM and inference scheduler endpoints.
Distributed Inference with llm-d Monitoring is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
9.1. Metrics for llm-d inference deployments
Distributed Inference with llm-d deployments expose Prometheus metrics that you can use to monitor inference performance, track resource usage, and detect operational issues. On OpenShift, the controller automatically creates PodMonitor and ServiceMonitor resources to collect these metrics when you deploy an LLMInferenceService. On other Kubernetes platforms, you must configure Prometheus scrape targets manually to collect metrics from the vLLM engine and inference scheduler endpoints. You can build custom dashboards for SLO compliance and capacity planning.
llm-d components expose three distinct categories of Prometheus metrics:
- vLLM engine metrics
-
The vLLM model servers expose metrics on port
8000at the/metricsendpoint. These metrics cover token throughput, request latency distributions, KV cache utilization, request queue depth, and model loading status. vLLM metrics provide the foundation for understanding individual model server performance and resource consumption. - EPP and inference scheduler metrics
-
The Endpoint Picker (EPP) exposes metrics at the
/metricsendpoint on themetricsservice port. These metrics cover request routing decisions, endpoint scoring, scheduling latency, plugin processing times, and prefix cache indexing performance. EPP metrics help you understand how requests are distributed across model server replicas and how efficiently the scheduler routes traffic. - Inference objective metrics
- The inference scheduler also exposes higher-level metrics that track end-to-end inference performance against service level objectives. These metrics include request error rates, time to first token (TTFT), time per output token (TPOT), SLO violation counts, and request duration distributions. Inference objective metrics provide the basis for SLO-based alerting and compliance monitoring.
These three metric categories work together to provide a complete picture of your llm-d deployment health. Use vLLM metrics to monitor individual model server capacity, EPP metrics to evaluate routing efficiency, and inference objective metrics to track overall service quality.
9.2. Verify metrics collection for llm-d deployments
On OpenShift, the controller automatically creates PodMonitor and ServiceMonitor resources to scrape metrics from vLLM engines and the inference scheduler when you deploy an LLMInferenceService. On other Kubernetes platforms, the controller does not create these resources. You must configure Prometheus scrape targets manually to collect metrics.
The following scrape targets expose llm-d metrics:
- vLLM engine metrics
-
Exposed on port
8000at the/metricsendpoint on each model server pod. - EPP and inference scheduler metrics
-
Exposed at the
/metricsendpoint on themetricsservice port of the scheduler service.
Prerequisites
- You have a deployed Distributed Inference with llm-d inference deployment on Red Hat OpenShift AI 3.4.
- You have cluster administrator access.
You have a Prometheus-based monitoring stack configured in your cluster.
On OpenShift, enable user workload monitoring. The Prometheus Operator is included automatically. See Configuring user workload monitoring.
On other Kubernetes platforms, you must have the Prometheus Operator installed if you want to use
PodMonitorandServiceMonitorresources, or a standalone Prometheus deployment configured with static or service-discovery-based scrape targets.
Procedure
On OpenShift:
Verify that the
PodMonitorresource for vLLM engine metrics exists in your deployment namespace:$ kubectl get podmonitors -n <NAMESPACE> -l app.kubernetes.io/component=llm-monitoringReplace
<NAMESPACE>with the namespace where your llm-d deployment runs.The output shows a
PodMonitorresource with a name that begins withkserve-llm-isvc-vllm-engine. This resource scrapes vLLM metrics from port8000on all model server pods.Verify that the
ServiceMonitorresource for inference scheduler metrics exists in your deployment namespace:$ kubectl get servicemonitors -n <NAMESPACE> -l app.kubernetes.io/component=llm-monitoringThe output shows a
ServiceMonitorresource with a name that begins withkserve-llm-isvc-scheduler. This resource scrapes EPP and inference scheduler metrics from themetricsport on the scheduler service.
On other Kubernetes platforms without the Prometheus Operator:
Configure your Prometheus instance to scrape the following targets in your deployment namespace:
Expand Table 9.1. llm-d scrape targets Component Port Path Pod selector vLLM engine
8000/metricsPods with the label
app: <POOL_NAME>, where<POOL_NAME>is the name of yourInferencePoolresource.EPP and inference scheduler
9090/metricsThe EPP service, selected by the label
app: <EPP_NAME>, where<EPP_NAME>is the name of your EPP deployment.
On all platforms:
Verify that Prometheus is scraping vLLM metrics by running the following PromQL query against your Prometheus instance:
vllm:num_requests_running{namespace="<NAMESPACE>"}Replace
<NAMESPACE>with your deployment namespace.On OpenShift, run this query from the web console by navigating to Observe > Metrics.
On other Kubernetes platforms, run this query by using your Prometheus web UI or API.
If the query returns results, Prometheus is successfully scraping vLLM metrics.
Verify that Prometheus is scraping EPP and inference scheduler metrics by running the following PromQL query:
llm_d_inference_scheduler_disagg_decision_total{namespace="<NAMESPACE>"}Replace
<NAMESPACE>with your deployment namespace.If the query returns results, Prometheus is successfully scraping inference scheduler metrics.
Metrics might take up to 60 seconds to appear after the LLMInferenceService is deployed. If metrics do not appear, verify that your monitoring stack is configured to scrape metrics from the deployment namespace. On OpenShift, verify that user workload monitoring is enabled and that PodMonitor and ServiceMonitor resources are present. On other platforms, verify that the Prometheus scrape configuration targets the correct namespace, ports, and label selectors.
9.3. PromQL queries for llm-d monitoring
You can use the following PromQL queries to monitor your Distributed Inference with llm-d deployments. The queries are organized into two groups: immediate failure and saturation indicators that drive operational alerts, and diagnostic drill-down queries that support deeper investigation.
Failure and saturation indicators
Use the following queries for dashboards and alerts that track service health and capacity limits.
| Operational concern | PromQL query |
|---|---|
| Overall error rate |
|
| Per-model error rate |
|
| Error rate by error code |
|
| Request rate by model |
|
| Overall latency P99 |
|
| Overall latency P90 |
|
| Model-specific TTFT P99 |
|
| Model-specific TPOT P99 |
|
| EPP end-to-end latency P99 |
|
| Plugin processing latency P99 |
|
| Scheduler health |
|
| Request preemptions per vLLM instance |
|
Diagnostic drill-down queries
Use the following queries for deeper investigation when failure and saturation indicators signal issues.
| Operational concern | PromQL query |
|---|---|
| KV cache utilization by pod |
|
| Request queue length by pod |
|
| Model throughput in tokens per second |
|
| Generation token rate by pod |
|
| Active requests per pod |
|
| Operational concern | PromQL query |
|---|---|
| Request distribution per instance |
|
| Token distribution across pods |
|
| Routing decision latency P99 |
|
| Operational concern | PromQL query |
|---|---|
| Prefix cache hit rate |
|
| Per-instance prefix cache hit rate |
|
| KV cache utilization percentage |
|
| EPP prefix indexer size |
|
| EPP prefix indexer hit ratio P90 |
|
| Operational concern | PromQL query |
|---|---|
| Prefill worker utilization |
|
| Decode worker KV cache utilization |
|
| Prefill queue length |
|
| Disaggregation decision rate by type |
|
| Disaggregation decision ratio |
|
9.4. Import Grafana dashboards for llm-d
You can import community Grafana dashboards to get a baseline monitoring view of your Distributed Inference with llm-d deployments. The dashboards provide pre-built visualizations for vLLM performance, failure and saturation indicators, diagnostic drill-downs, KV cache performance, and prefill/decode disaggregation metrics.
Prerequisites
- You have a Grafana instance deployed and connected to a Prometheus data source that collects llm-d metrics. Grafana is not included with OpenShift. You must deploy Grafana separately, for example, by using the Grafana community Operator.
- Metrics collection is enabled for your Distributed Inference with llm-d deployment.
Procedure
Download the community Grafana dashboard JSON files from the llm-d project.
The following dashboards are available:
- llm-d vLLM Overview
- General vLLM metrics overview for monitoring llm-d inference servers. Download from llm-d-vllm-overview.json.
- llm-d Failure and Saturation Indicators
- Key failure and saturation indicators for identifying system issues and capacity constraints. Download from llm-d-failure-saturation-dashboard.json.
- llm-d Diagnostic Drill-Down
- Detailed diagnostic metrics for investigating performance issues. Download from llm-d-diagnostic-drilldown-dashboard.json.
- llm-d Performance Dashboard
- Performance metrics including KV cache utilization. Download from llm-performance-kv-cache.json.
- P/D Coordinator Metrics
- Prefill/decode disaggregation performance metrics, including vLLM end-to-end latency, prefill duration, decode duration, and phase breakdown. Download from pd-coordinator-metrics.json.
- In the Grafana web interface, navigate to Dashboards > New > Import.
- Upload a dashboard JSON file or paste the JSON content into the import dialog.
- Select the Prometheus data source that collects metrics from your llm-d deployment.
- Click Import to create the dashboard.
- Repeat the import process for each dashboard JSON file.
Verification
- Navigate to a newly imported dashboard and verify that the panels display data from your llm-d deployment. If a "No data" message is shown, verify that the Prometheus data source is correctly configured and that metrics collection is enabled for the llm-d namespace.
In disconnected environments, download the dashboard JSON files from a connected network and transfer them to a workstation with access to the Grafana instance. The dashboards do not reference external data sources, so they function in disconnected environments after Prometheus and Grafana are available.
9.5. vLLM metrics for llm-d
The vLLM model servers in Distributed Inference with llm-d deployments expose Prometheus metrics on port 8000 at the /metrics endpoint. You can use these metrics to monitor token throughput, request latency, KV cache usage, and request queue depth for individual model server replicas.
Token throughput metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| counter |
| The total number of prompt tokens processed by the model server. |
|
| counter |
| The total number of generation tokens produced by the model server. |
|
| histogram |
| The end-to-end request latency distribution in seconds, measured from request receipt to response completion. |
Latency metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| histogram |
| The time to first token (TTFT) distribution in seconds. This measures the latency from request submission to the first generated token. |
|
| histogram |
| The inter-token latency distribution in seconds. This measures the time between consecutive output tokens. |
KV cache metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| gauge |
|
The GPU KV cache utilization as a percentage between |
|
| counter |
| The total number of request preemptions caused by KV cache pressure. Preemptions occur when the cache is full and running requests must be evicted to make room for new requests. |
Request queue metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| gauge |
| The current number of requests being actively processed by the model server. |
|
| gauge |
| The current number of requests waiting in the queue for processing. |
|
| counter |
| The total number of successfully completed inference requests, broken down by completion reason. |
Prefix cache metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| counter |
| The total number of prefix cache hits. A cache hit occurs when a request prompt matches cached key-value data, avoiding redundant computation. |
|
| counter |
|
The total number of prefix cache queries. Use this metric with |
9.6. EPP and inference scheduler metrics for llm-d
The Endpoint Picker (EPP) and inference scheduler in Distributed Inference with llm-d deployments expose Prometheus metrics at the /metrics endpoint on the metrics service port. You can use these metrics to monitor request routing decisions, scheduling latency, inference objective compliance, and prefix cache indexing performance.
Inference objective metrics
The inference objective metrics track end-to-end inference performance and SLO compliance. These metrics use the inference_objective_ prefix.
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| counter |
| The total number of inference requests processed by the scheduler. |
|
| counter |
| The total number of inference request errors, broken down by error code. |
|
| histogram |
| The end-to-end inference request duration distribution in seconds. |
|
| histogram |
| The time to first token (TTFT) distribution in seconds. |
|
| histogram |
| The time per output token (TPOT) distribution in seconds. |
|
| histogram |
| The predicted TTFT distribution in seconds, used by the scheduler for routing decisions. |
|
| histogram |
| The predicted TPOT distribution in seconds, used by the scheduler for routing decisions. |
|
| counter |
|
The total number of SLO violations, broken down by violation type: |
|
| gauge |
| The current number of active inference requests. |
|
| histogram |
| The input token count distribution per request. |
|
| histogram |
| The output token count distribution per request. |
Scheduling and routing metrics
The scheduling and routing metrics track the internal performance of the Endpoint Picker. These metrics use the inference_extension_ prefix.
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| histogram | None | The end-to-end scheduling duration in seconds, measured from request receipt to endpoint selection. |
|
| counter |
|
The total number of scheduling attempts, broken down by status: |
|
| histogram |
| The plugin processing duration in seconds, broken down by extension point, plugin type, and plugin name. Use this metric to identify slow plugins that impact scheduling latency. |
|
| counter |
| The total number of model rewrite decisions made by the scheduler. |
Prefix cache indexer metrics
The prefix cache indexer metrics track KV cache prefix matching performance in the scheduler. These metrics use the inference_extension_ prefix.
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| gauge | None | The current size of the prefix cache index maintained by the scheduler. |
|
| histogram | None | The prefix cache hit ratio distribution. A higher hit ratio indicates that requests frequently match cached prefixes, reducing redundant computation. |
|
| histogram | None | The prefix cache hit size distribution in bytes. Larger hit sizes indicate more effective prefix reuse. |
Inference pool metrics
The inference pool metrics track aggregate statistics across all endpoints in a pool. These metrics use the inference_pool_ prefix.
| Metric name | Type | Labels | Description |
|---|---|---|---|
|
| gauge |
| The average KV cache utilization across all endpoints in the inference pool. |
|
| gauge |
| The average request queue size across all endpoints in the inference pool. |
|
| gauge |
| The average number of running requests across all endpoints in the inference pool. |
|
| gauge |
| The number of ready pods in the inference pool. |
|
| gauge |
| The request queue size for a specific pod in the inference pool. |
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}