Chapter 10. Distributed Inference with llm-d observability

Procedure

1. Install the Red Hat build of Tempo Operator:

   1. On OpenShift 4.20 and later, install from the software catalog

      • On OpenShift 4.19, install from the OperatorHub.
   2. Search for Red Hat build of Tempo.
   3. Select the Operator from the redhat-operators catalog.
   4. Click Install.
   5. On the Install Operator page, accept the default settings and click Install.

2. Verify the Tempo Operator installation:

   $ oc get csv -n openshift-operators | grep tempo

Procedure

1. Create a namespace for the Tempo deployment:

   $ oc new-project tempo-system

2. Save the following YAML to a file named tempo-monolithic.yaml. Create a TempoMonolithic custom resource:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoMonolithic
   metadata:
     name: tempo-llmd
     namespace: tempo-system
   spec:
     storage:
       traces:
         backend: memory
     jaegerui:
       enabled: true
       route:
         enabled: true
     ingestion:
       otlp:
         grpc:
           enabled: true
         http:
           enabled: true

   where:

   storage.traces.backend

   Specifies the trace storage backend. The value memory provides temporary, in-memory storage for development and testing.

   jaegerui.enabled

   Enables the Jaeger Query Frontend for trace visualization.

   jaegerui.route.enabled

   Creates a Route for external access to the Jaeger user interface.

   ingestion.otlp.grpc.enabled

   Enables ingestion of telemetry data by using OTLP over gRPC.

   ingestion.otlp.http.enabled

   Enables ingestion of telemetry data by using OTLP over HTTP.

   Note

   The memory backend does not persist traces after the Tempo pod restarts. For production environments, configure a supported persistent storage backend.

3. Apply the TempoMonolithic custom resource:

   $ oc apply -f tempo-monolithic.yaml

4. Wait for the Tempo instance to be ready:

   $ oc get tempomonolithic tempo-llmd -n tempo-system -w

   Wait until the READY status shows True.

5. Retrieve the Jaeger user interface route URL:

   $ oc get route -n tempo-system

   Example output

   NAME                         HOST/PORT
   tempo-llmd-jaegerui          tempo-llmd-jaegerui-tempo-system.apps.example.com

Verification

1. Verify that the TempoMonolithic custom resource shows READY=True:

   $ oc get tempomonolithic tempo-llmd -n tempo-system

   The output shows READY status as True.

2. Verify that Tempo pods are running in the tempo-system namespace:

   $ oc get pods -n tempo-system

   All pods show Running status.

3. Verify that the Jaeger user interface route is accessible:

   $ oc get route tempo-llmd-jaegerui -n tempo-system -o jsonpath='{.spec.host}'

   The output shows the route hostname.

Procedure

1. Save the following YAML to a file named llminferenceservice-tracing.yaml. Create an LLMInferenceService custom resource with distributed tracing enabled:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: traced-llm-inference
     namespace: llm-inference
   spec:
     tracing:
       exporterEndpoint: "http://tempo-tracing:4317"
       sampler: "parentbased_traceidratio"
       samplerArg: "0.05"
       exporter: "otlp"
     model:
       uri: hf://Qwen/Qwen2.5-7B-Instruct
       name: Qwen/Qwen2.5-7B-Instruct
     replicas: 1
     router:
       scheduler: {}
       route: {}
       gateway: {}
     template:
       containers:
       - name: main
         image: registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.5.0-ea.1-1780065492
         imagePullPolicy: Always
         resources:
           limits:
             cpu: "4"
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: "2"
             memory: 16Gi
             nvidia.com/gpu: "1"

   where:

   spec.tracing.exporterEndpoint

   Specifies the Tempo endpoint for trace export. Use the cluster-internal service DNS name tempo-tracing:4317.

   spec.tracing.sampler

   Specifies the sampling strategy. Use parentbased_traceidratio for probabilistic sampling.

   spec.tracing.samplerArg

   Specifies the sampling rate as a ratio between 0 and 1. A value of 0.05 captures 5% of traces. For production deployments, use 1% to 5% to reduce trace volume and storage costs.

   spec.tracing.exporter

   Specifies the trace export protocol. Use otlp for OTLP over gRPC.

2. Apply the LLMInferenceService custom resource:

   $ oc apply -f llminferenceservice-tracing.yaml

3. Wait for the LLMInferenceService resource to be ready:

   $ oc get llmisvc traced-llm-inference -n llm-inference -w

   Wait until READY=True.

4. Verify that the tracing configuration is applied:

   $ oc get llmisvc traced-llm-inference -n llm-inference -o jsonpath='{.spec.tracing}'

Verification

1. Verify that the LLMInferenceService resource shows READY=True:

   $ oc get llmisvc traced-llm-inference -n llm-inference

2. Verify that the scheduler and vLLM pods are running:

   $ oc get pods -n llm-inference

   All pods show Running status.

Procedure

1. Choose a sampling strategy for your environment:

   • Development and testing: Use parentbased_always_on to sample all traces.
   • Production: Use parentbased_traceidratio with an initial sampling rate of 1% to 5%.

2. Save the following YAML to a file named llminferenceservice-tracing.yaml. Update the LLMInferenceService resource to use probabilistic sampling. The following example configures a 5% sampling rate:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: traced-llm-inference
     namespace: llm-inference
   spec:
     tracing:
       exporterEndpoint: "http://tempo-tracing:4317"
       sampler: "parentbased_traceidratio"
       samplerArg: "0.05"
       exporter: "otlp"

3. Apply the configuration:

   $ oc apply -f llminferenceservice-tracing.yaml

4. Monitor trace volume and adjust the sampling rate as needed:

   1. Check Tempo storage usage:

      $ oc exec -n tempo-system <tempo_pod> -- df -h

      where:

      __<tempo_pod>__

      Specifies the name of the Tempo pod.

   2. In the Jaeger UI, review the number and distribution of sampled traces.
   3. Adjust the samplerArg value based on observed trace volume.

Verification

1. Confirm that the sampling configuration is present in the LLMInferenceService resource:

   $ oc get llminferenceservice traced-llm-inference -n llm-inference -o yaml

2. Verify that the spec.tracing.sampler field is set to parentbased_traceidratio and that the spec.tracing.samplerArg field is set to "0.05".
3. Generate inference requests and confirm that traces appear in the tracing interface at approximately the configured sampling rate.

Procedure

1. Send a test inference request to generate trace data:

   $ SERVICE_URL=$(oc get llmisvc traced-llm-inference -n llm-inference \
       -o jsonpath='{.status.url}')
   
   $ curl -X POST "${SERVICE_URL}/v1/chat/completions" \
       -H "Content-Type: application/json" \
       -d '{
         "model": "Qwen/Qwen2.5-7B-Instruct",
         "messages": [{"role": "user", "content": "What is Kubernetes?"}],
         "max_tokens": 100
       }'

   Note

   If you configured trace sampling, send multiple requests to ensure at least one trace is captured. The number of requests needed is 1 / sample_rate. For example, with a 5% sample rate, send at least 20 requests to see a trace in Jaeger UI.

2. Access the Jaeger UI by using port forwarding:

   $ oc port-forward -n tempo-system svc/tempo-llmd-query-frontend 16686:16686

3. Open a web browser to http://localhost:16686.

4. Search for traces:

   1. In the Service dropdown, select inference-server-decode or gateway-api-inference-extension.
   2. Set the Lookback time to Last 15 minutes.
   3. Click Find Traces.

5. Examine a trace to verify end-to-end spans:

   1. Click a trace from the search results.

   2. Verify the trace has the following spans:

      • Scheduler span showing request routing
      • vLLM span showing model inference
   3. Check that all spans share the same Trace ID.
   4. Verify span attributes include service names.

6. Verify trace context propagation:

   1. Expand each span in the trace view.
   2. Verify parent-child relationships between spans.
   3. Confirm span timing data is accurate.