Chapter 11. Managing observability

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the OpenShift console, click Operators Installed Operators.
3. Search for the Red Hat OpenShift AI Operator, and then click the Operator name to open the Operator details page.
4. Click the DSCInitialization tab.
5. Click the default instance name (for example, default-dsci) to open the instance details page.
6. Click the YAML tab to show the instance specifications.

7. In the spec.monitoring section, set the value of the managementState field to Managed, and configure metrics, alerting, and tracing settings as shown in the following example:

   Example monitoring configuration

   # ...
   spec:
     monitoring:
       managementState: Managed                 # Required: Enables and manages the observability stack
       namespace: redhat-ods-monitoring    # Required: Namespace where monitoring components are deployed
       alerting: {}                              # Alertmanager configuration, uses default settings if empty
       metrics:                                  # Prometheus configuration for metrics collection
         replicas: 1                             # Optional: Number of Prometheus instances
         resources:                              # CPU and memory requests and limits for Prometheus pods
           cpulimit: 500m                        # Optional: Maximum CPU allocation in millicores
           cpurequest: 100m                      # Optional: Minimum CPU allocation in millicores
           memorylimit: 512Mi                    # Optional: Maximum memory allocation in mebibytes
           memoryrequest: 256Mi                  # Optional: Minimum memory allocation in mebibytes
         storage:                                # Storage configuration for metrics data
           size: 5Gi                             # Required: Storage size for Prometheus data
           retention: 90d                        # Required: Retention period for metrics data in days
         exporters: {}                           # External metrics exporters
       traces:                                   # Tempo backend for distributed tracing
         sampleRatio: '0.1'                      # Optional: Portion of traces to sample, expressed as a decimal
         storage:                                # Storage configuration for trace data
           backend: pv                           # Required: Storage backend for Tempo traces (pv, s3, or gcs)
           retention: 2160h                      # Optional: Retention period for trace data in hours
         exporters: {}                           # External traces exporters
   # ...

   Copy to Clipboard Toggle word wrap

8. Click Save to apply your changes.

1. In the OpenShift web console, click Workloads Pods.
2. From the project list, select redhat-ods-monitoring.

3. Confirm that there are running pods for your configuration. The following pods indicate that the observability stack is active:

   alertmanager-data-science-monitoringstack-#      2/2   Running   0   1m
   data-science-collector-collector-#               1/1   Running   0   1m
   prometheus-data-science-monitoringstack-#        2/2   Running   0   1m
   tempo-data-science-tempomonolithic-#             1/1   Running   0   1m
   thanos-querier-data-science-thanos-querier-#     2/2   Running   0   1m

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift web console as a cluster administrator or project owner.
2. Click Workloads Deployments.
3. In the Project list at the top of the page, select the project where your workload is deployed.
4. Identify the deployment that you want to collect metrics from and click its name.
5. On the Deployment details page, click the YAML tab.

6. In the YAML editor, add the required label under the spec.template.metadata.labels section, as shown in the following example:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: <example_name>
     namespace: <example_namespace>
   spec:
     template:
       metadata:
         labels:
           monitoring.opendatahub.io/scrape: 'true'
   # ...

   Copy to Clipboard Toggle word wrap

7. Click Save.

   OpenShift automatically rolls out a new ReplicaSet and pods with the updated label. When the new pods start, the observability stack begins scraping their metrics.

1. Access Prometheus by using a route:

   1. In the OpenShift web console, click Networking Routes.
   2. From the project list, select redhat-ods-monitoring.
   3. Locate the route associated with the Prometheus service, such as data-science-prometheus-route.
   4. Click the Location URL to open the Prometheus web console.

2. Alternatively, access Prometheus locally by using port forwarding:

   1. List the Prometheus pods:

      $ oc get pods -n redhat-ods-monitoring -l prometheus=data-science-monitoringstack

      Copy to Clipboard Toggle word wrap

   2. Start port forwarding:

      $ oc port-forward __<prometheus-pod-name>__ 9090:9090 -n redhat-ods-monitoring

      Copy to Clipboard Toggle word wrap

   3. In a web browser, open the following URL:

      http://localhost:9090

      Copy to Clipboard Toggle word wrap

3. In the Prometheus web console, search for a metric exposed by your workload.

   If the label is applied correctly and the workload exposes metrics, the metrics appear in the Prometheus instance deployed by OpenShift AI.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. Click Operators Installed Operators.
3. Select the Red Hat OpenShift AI Operator from the list.
4. Click the DSCInitialization tab.
5. Click the default DSCI instance, for example, default-dsci, to open its details page.
6. Click the YAML tab.

7. In the spec.monitoring.metrics section, add an exporters list that defines the external receiver configuration, as shown in the following example:

   spec:
     monitoring:
       metrics:
         exporters:
           - name: <external_exporter_name>
             type: <type>
             endpoint: https://example-otlp-receiver.example.com:4317

   Copy to Clipboard Toggle word wrap
   • name: A unique, descriptive name for the exporter configuration. Do not use reserved names such as prometheus or otlp/tempo.

   • type: The protocol used for export, for example:

     • otlp: For OpenTelemetry-compatible backends using gRPC or HTTP.
     • prometheusremotewrite: For Prometheus-compatible systems that use the remote write protocol.

   • endpoint: The full URL of your external metrics receiver. For OTLP, endpoints typically use port 4317 (gRPC) or 4318 (HTTP). For Prometheus remote write, endpoints typically end with /api/v1/write. For example:

     • otlp: https://example-otlp-receiver.example.com:4317 (gRPC) or https://example-otlp-receiver.example.com:4318 (HTTP)
     • prometheusremotewrite: https://example-prometheus-remote.example.com/api/v1/write

8. Click Save.

   The OpenTelemetry Collector automatically reloads its configuration and begins forwarding metrics to the specified external endpoint.

Verification

1. Verify that the OpenTelemetry Collector pods restart and apply the new configuration:

   $ oc get pods -n redhat-ods-monitoring

   Copy to Clipboard Toggle word wrap

   The data-science-collector-collector-* pods should restart and display a Running status.

2. In your external observability platform, verify that new metrics from OpenShift AI appear in the metrics list or dashboard.

Procedure

1. Find the OpenTelemetry Collector endpoint.

   The OpenTelemetry Collector receives trace data from instrumented applications by using the OpenTelemetry Protocol (OTLP).

   1. In the OpenShift web console, navigate to Networking Services.
   2. In the Project list, select the monitoring namespace, for example, {monitoring-default-namespace}.
   3. Locate the Service named data-science-collector or a similar name associated with the OpenTelemetry Collector.

   4. Use the Service name or ClusterIP as the OTLP endpoint in your application configuration.

      Your application must export traces to one of the following ports on the collector service:

      • gRPC: 4317

      • HTTP: 4318

        Example environment variable:

        OTEL_EXPORTER_OTLP_ENDPOINT=http://data-science-collector.redhat-ods-monitoring.svc.cluster.local:4318

        Copy to Clipboard Toggle word wrap
        Note

        See the Red Hat build of OpenTelemetry documentation for details about configuring application instrumentation.

2. Connect your visualization tool to the Tempo query service.

   You can use a visualization tool, such as Grafana or Jaeger, to query and display traces from the Red Hat build of Tempo deployed by OpenShift AI.

   1. In the OpenShift web console, navigate to Networking Services.
   2. In the Project list, select the monitoring namespace, for example, {monitoring-default-namespace}.
   3. Locate the Service named tempo-query or tempo-query-frontend.

   4. To make the service accessible to external tools, a cluster administrator must perform one of the following actions:

      • Create a route: Expose the Tempo Query service externally by creating an OpenShift route.

      • Use port forwarding: Temporarily forward a local port to the Tempo Query service by using the OpenShift CLI (oc):

        $ oc port-forward svc/tempo-query-frontend 3200:3200 -n redhat-ods-monitoring

        Copy to Clipboard Toggle word wrap

        After the port is forwarded, connect your visualization tool to the Tempo Query API endpoint, for example:

        http://localhost:3200

        Copy to Clipboard Toggle word wrap
        Note

        See the Tempo Operator documentation for details about connecting to Tempo.

Verification

1. Confirm that your instrumented application is generating and exporting trace data.

2. Verify that the OpenTelemetry Collector pod is running in the monitoring namespace:

   $ oc get pods -n redhat-ods-monitoring | grep collector

   Copy to Clipboard Toggle word wrap

   The data-science-collector-collector-* pod should display a Running status.

3. Access your visualization tool and confirm that new traces appear in the trace list or search view.

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as a cluster administrator:

   $ oc login https://api.198.51.100.10:6443

   Copy to Clipboard Toggle word wrap

2. Verify that the Alertmanager pods are running in the monitoring namespace:

   $ oc get pods -n redhat-ods-monitoring | grep alertmanager

   Copy to Clipboard Toggle word wrap

   Example output:

   alertmanager-data-science-monitoringstack-0   2/2   Running   0   2h
   alertmanager-data-science-monitoringstack-1   2/2   Running   0   2h

   Copy to Clipboard Toggle word wrap

3. Confirm that a ClusterIP service exposes the Alertmanager web interface on port 9093:

   $ oc get svc -n redhat-ods-monitoring | grep alertmanager

   Copy to Clipboard Toggle word wrap

   Example output:

   data-science-monitoringstack-alertmanager     ClusterIP   198.51.100.5   <none>   9093/TCP

   Copy to Clipboard Toggle word wrap

4. Start a local port forward to the Alertmanager service:

   $ oc port-forward svc/data-science-monitoringstack-alertmanager 9093:9093 -n redhat-ods-monitoring

   Copy to Clipboard Toggle word wrap

5. In a web browser, open the following URL to access the Alertmanager web interface:

   http://localhost:9093

   Copy to Clipboard Toggle word wrap