Chapter 1. Configuring distributed tracing

1. Component override: If you set spec.template.<component>.resources, the Operator uses these values.
2. Size profile: If you set spec.size, the Operator uses the t-shirt size profile values.
3. Total resources: If you set spec.resources.total, the Operator uses the percentage-based split.
4. No resources: If you do not set any of the previous options, the Operator does not apply resource requests.

Procedure

1. In the OpenTelemetryCollector custom resource of the OpenTelemetry Collector, enable the Span Metrics Connector, which derives metrics from traces and exports the metrics in the Prometheus format.

   The following OpenTelemetryCollector custom resource example configures span RED:

   apiVersion: opentelemetry.io/v1beta1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
   spec:
     mode: deployment
     observability:
       metrics:
         enableMetrics: true
     config: |
       connectors:
         spanmetrics:
           metrics_flush_interval: 15s
   
       receivers:
         otlp:
           protocols:
             grpc:
             http:
   
       exporters:
         prometheus:
           endpoint: 0.0.0.0:8889
           add_metric_suffixes: false
           resource_to_telemetry_conversion:
             enabled: true
   
         otlp/traces:
           auth:
             authenticator: bearertokenauth
           endpoint: tempo-redmetrics-gateway.mynamespace.svc.cluster.local:8090
           headers:
             X-Scope-OrgID: dev
           tls:
             ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
             insecure: false
   
       extensions:
         bearertokenauth:
           filename: /var/run/secrets/kubernetes.io/serviceaccount/token
   
       service:
         extensions:
         - bearertokenauth
         pipelines:
           traces:
             receivers: [otlp]
             exporters: [otlp/traces, spanmetrics]
           metrics:
             receivers: [spanmetrics]
             exporters: [prometheus]
   
   # ...

   where:

   enableMetrics

   Creates the ServiceMonitor custom resource to enable scraping of the Prometheus exporter.

   spanmetrics

   The Span Metrics Connector receives traces and exports metrics.

   otlp

   The OTLP receiver receives spans in the OpenTelemetry protocol.

   prometheus

   The Prometheus Exporter exports metrics in the Prometheus format.

   enabled

   Resource attributes are dropped by default.

   exporters

   The Span Metrics Connector is configured as an exporter in the traces pipeline.

   receivers

   The Span Metrics Connector is configured as a receiver in the metrics pipeline.

2. In the TempoStack custom resource, enable the Monitor tab and set the Prometheus endpoint to the Thanos querier service to query the data from your user-defined monitoring stack.

   The following TempoStack custom resource example enables the Monitor tab:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: redmetrics
   spec:
     storage:
       secret:
         name: minio-test
         type: s3
     storageSize: 1Gi
     tenants:
       mode: openshift
       authentication:
         - tenantName: dev
           tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
     template:
       gateway:
         enabled: true
       queryFrontend:
         jaegerQuery:
           monitorTab:
             enabled: true
             prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
             redMetricsNamespace: ""
   
   # ...

   where:

   enabled

   Enables the monitoring tab in the Jaeger console.

   prometheusEndpoint

   Service name for Thanos Querier from user-workload monitoring.

3. Optional: Use the span RED metrics that the Span Metrics Connector generates with alerting rules.

   For example, for alerts about a slow service or to define service level objectives (SLOs), the connector creates a traces_span_metrics_duration_bucket histogram and the traces_span_metrics_calls counter metric. The following example PrometheusRule custom resource defines an alerting rule for SLO when not serving 95% of requests within 2000 ms on the front-end service:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: span-red
   spec:
     groups:
     - name: server-side-latency
       rules:
       - alert: SpanREDFrontendAPIRequestLatency
         expr: histogram_quantile(0.95, sum(rate(traces_span_metrics_duration_bucket{service_name="frontend", span_kind="SPAN_KIND_SERVER"}[5m])) by (le, service_name, span_name)) > 2000
         labels:
           severity: Warning
         annotations:
           summary: "High request latency on {{$labels.service_name}} and {{$labels.span_name}}"
           description: "{{$labels.instance}} has 95th request latency above 2s (current value: {{$value}}s)"

   where:

   expr

   Expression for checking if 95% of the front-end server response time values are below 2000 ms. The time range ([5m]) must be at least four times the scrape interval and long enough to accommodate a change in the metric.

Procedure

1. Enable multitenancy and query RBAC in the TempoStack custom resource (CR), for example:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: simplest
     namespace: chainsaw-multitenancy
   spec:
     storage:
       secret:
         name: minio
         type: s3
     storageSize: 1Gi
     resources:
       total:
         limits:
           memory: 2Gi
           cpu: 2000m
     tenants:
       mode: openshift
       authentication:
         - tenantName: dev
           tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
     template:
       gateway:
         enabled: true
         rbac:
           enabled: true
       queryFrontend:
         jaegerQuery:
           enabled: false

   where:

   gateway.enabled

   Always set to true.

   rbac.enabled

   Always set to true.

   jaegerQuery.enabled

   Always set to false.

2. Create a cluster role and cluster role binding to grant the target users the permissions to access the tenant that you specified in the TempoStack CR, for example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: tempo-dev-read
   rules:
   - apiGroups: [tempo.grafana.com]
     resources: [dev]
     resourceNames: [traces]
     verbs: [get]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: tempo-dev-read
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: tempo-dev-read
   subjects:
     - kind: Group
       apiGroup: rbac.authorization.k8s.io
       name: system:authenticated

   where:

   resources

   Tenant name in the TempoStack CR.

   name

   All authenticated OpenShift users.

3. Grant the target users the permissions to read attributes for the project. You can do this by running the following command:

   $ oc adm policy add-role-to-user view <username> -n <project>