Chapter 3. Distributed tracing platform (Tempo)

Procedure

1. Go to Operators OperatorHub and search for Tempo Operator.

2. Select the Tempo Operator that is provided by Red Hat.

   Important

   The following selections are the default presets for this Operator:

   • Update channel stable
   • Installation mode All namespaces on the cluster
   • Installed Namespace openshift-tempo-operator
   • Update approval Automatic
3. Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
4. Select Install Install View Operator.

Procedure

1. Create a project for the Tempo Operator by running the following command:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     labels:
       kubernetes.io/metadata.name: openshift-tempo-operator
       openshift.io/cluster-monitoring: "true"
     name: openshift-tempo-operator
   EOF

2. Create an Operator group by running the following command:

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-tempo-operator
     namespace: openshift-tempo-operator
   spec:
     upgradeStrategy: Default
   EOF

3. Create a subscription by running the following command:

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: tempo-product
     namespace: openshift-tempo-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: tempo-product
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

Procedure

1. Go to Home Projects Create Project to create a project of your choice for the TempoStack instance that you will create in a subsequent step.

2. Go to Workloads Secrets Create From YAML to create a secret for your object storage bucket in the project that you created for the TempoStack instance. For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoStack instance.

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Go to Operators Installed Operators.
   2. Select TempoStack Create TempoStack YAML view.

   3. In the YAML view, customize the TempoStack custom resource (CR):

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: sample
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: <value>Gi 1
        storage:
          secret: 2
            name: <secret_name> 3
            type: <secret_provider> 4
          tls: 5
            enabled: true
            caName: <ca_certificate_configmap_name> 6
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route
        resources: 7
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m

      1

      Size of the persistent volume claim for the Tempo WAL. The default is 10Gi.

      2

      Secret you created in step 2 for the object storage that had been set up as one of the prerequisites.

      3

      Value of the name in the metadata of the secret.

      4

      Accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or {odf-full}.

      5

      Optional.

      6

      Optional: Name of a ConfigMap object that contains a CA certificate.

      7

      Optional.

      Example of a TempoStack CR for AWS S3 and MinIO storage

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: simplest
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: 1Gi
        storage: 1
          secret:
            name: minio-test
            type: s3
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        template:
          queryFrontend:
            jaegerQuery: 2
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      1

      In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2.

      2

      The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the TempoStack instance.
2. Go to Operators Installed Operators to verify that the Status of the TempoStack instance is Condition: Ready.
3. Go to Workloads Pods to verify that all the component pods of the TempoStack instance are running.

4. Access the Tempo console:

   1. Go to Networking Routes and Ctrl+F to search for tempo.

   2. In the Location column, open the URL to access the Tempo console.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Run the following command to create a project of your choice for the TempoStack instance that you will create in a subsequent step:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: <project_of_tempostack_instance>
   EOF

2. In the project that you created for the TempoStack instance, create a secret for your object storage bucket by running the following command:

   $ oc apply -f - << EOF
   <object_storage_secret>
   EOF

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoStack instance in the project that you created for it:

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Customize the TempoStack custom resource (CR):

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: sample
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: <value>Gi 1
        storage:
          secret: 2
            name: <secret_name> 3
            type: <secret_provider> 4
          tls: 5
            enabled: true
            caName: <ca_certificate_configmap_name> 6
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route
        resources: 7
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m

      1

      Size of the persistent volume claim for the Tempo WAL. The default is 10Gi.

      2

      Secret you created in step 2 for the object storage that had been set up as one of the prerequisites.

      3

      Value of the name in the metadata of the secret.

      4

      Accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or {odf-full}.

      5

      Optional.

      6

      Optional: Name of a ConfigMap object that contains a CA certificate.

      7

      Optional.

      Example of a TempoStack CR for AWS S3 and MinIO storage

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: simplest
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: 1Gi
        storage: 1
          secret:
            name: minio-test
            type: s3
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        template:
          queryFrontend:
            jaegerQuery: 2
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      1

      In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2.

      2

      The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <tempostack_cr>
      EOF

Verification

1. Verify that the status of all TempoStack components is Running and the conditions are type: Ready by running the following command:

   $ oc get tempostacks.tempo.grafana.com simplest -o yaml

2. Verify that all the TempoStack component pods are running by running the following command:

   $ oc get pods

3. Access the Tempo console:

   1. Query the route details by running the following command:

      $ oc get route

   2. Open https://<route_from_previous_step> in a web browser.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Go to Home Projects Create Project to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step.

2. Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.

   Important

   Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.

   Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this in Workloads Secrets Create From YAML.

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoMonolithic instance:

   Note

   You can create multiple TempoMonolithic instances in separate projects on the same cluster.

   1. Go to Operators Installed Operators.
   2. Select TempoMonolithic Create TempoMonolithic YAML view.

   3. In the YAML view, customize the TempoMonolithic custom resource (CR).

      The following TempoMonolithic CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoMonolithic
      metadata:
        name: <metadata_name>
        namespace: <project_of_tempomonolithic_instance>
      spec:
        storage:
          traces:
            backend: <supported_storage_type> 1
            size: <value>Gi 2
            s3: 3
              secret: <secret_name> 4
          tls: 5
            enabled: true
            caName: <ca_certificate_configmap_name> 6
        jaegerui:
          enabled: true 7
          route:
            enabled: true 8
        resources: 9
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m

      1

      Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv. The accepted values for object storage are s3, gcs, or azure, depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down.

      2

      Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi. For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi. For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is 10Gi.

      3

      Optional: For object storage, the type of object storage. The accepted values are s3, gcs, and azure, depending on the used object store type.

      4

      Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup".

      5

      Optional.

      6

      Optional: Name of a ConfigMap object that contains a CA certificate.

      7

      Enables the Jaeger UI.

      8

      Enables creation of a route for the Jaeger UI.

      9

      Optional.

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the TempoMonolithic instance.
2. Go to Operators Installed Operators to verify that the Status of the TempoMonolithic instance is Condition: Ready.
3. Go to Workloads Pods to verify that the pod of the TempoMonolithic instance is running.

4. Access the Jaeger UI:

   1. Go to Networking Routes and Ctrl+F to search for jaegerui.

      Note

      The Jaeger UI uses the tempo-<metadata_name_of_TempoMonolithic_CR>-jaegerui route.

   2. In the Location column, open the URL to access the Jaeger UI.

5. When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_TempoMonolithic_CR>:4317 (OTLP/gRPC) and tempo-<metadata_name_of_TempoMonolithic_CR>:4318 (OTLP/HTTP) endpoints inside the cluster.

   The Tempo API is available at the tempo-<metadata_name_of_TempoMonolithic_CR>:3200 endpoint inside the cluster.

Procedure

1. Run the following command to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: <project_of_tempomonolithic_instance>
   EOF

2. Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.

   Important

   Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: {odf-full}, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.

   Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this by running the following command:

   $ oc apply -f - << EOF
   <object_storage_secret>
   EOF

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoMonolithic instance in the project that you created for it.

   Tip

   You can create multiple TempoMonolithic instances in separate projects on the same cluster.

   1. Customize the TempoMonolithic custom resource (CR).

      The following TempoMonolithic CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoMonolithic
      metadata:
        name: <metadata_name>
        namespace: <project_of_tempomonolithic_instance>
      spec:
        storage:
          traces:
            backend: <supported_storage_type> 1
            size: <value>Gi 2
            s3: 3
              secret: <secret_name> 4
          tls: 5
            enabled: true
            caName: <ca_certificate_configmap_name> 6
        jaegerui:
          enabled: true 7
          route:
            enabled: true 8
        resources: 9
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m

      1

      Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv. The accepted values for object storage are s3, gcs, or azure, depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down.

      2

      Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi. For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi. For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is 10Gi.

      3

      Optional: For object storage, the type of object storage. The accepted values are s3, gcs, and azure, depending on the used object store type.

      4

      Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup".

      5

      Optional.

      6

      Optional: Name of a ConfigMap object that contains a CA certificate.

      7

      Enables the Jaeger UI.

      8

      Enables creation of a route for the Jaeger UI.

      9

      Optional.

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <tempomonolithic_cr>
      EOF

Verification

1. Verify that the status of all TempoMonolithic components is Running and the conditions are type: Ready by running the following command:

   $ oc get tempomonolithic.tempo.grafana.com <metadata_name_of_tempomonolithic_cr> -o yaml

2. Run the following command to verify that the pod of the TempoMonolithic instance is running:

   $ oc get pods

3. Access the Jaeger UI:

   1. Query the route details for the tempo-<metadata_name_of_tempomonolithic_cr>-jaegerui route by running the following command:

      $ oc get route

   2. Open https://<route_from_previous_step> in a web browser.

4. When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_tempomonolithic_cr>:4317 (OTLP/gRPC) and tempo-<metadata_name_of_tempomonolithic_cr>:4318 (OTLP/HTTP) endpoints inside the cluster.

   The Tempo API is available at the tempo-<metadata_name_of_tempomonolithic_cr>:3200 endpoint inside the cluster.

Procedure

1. Create an AWS S3 bucket.

2. Create the following trust.json file for the AWS IAM policy that will set up a trust relationship for the AWS IAM role, created in the next step, with the service account of the TempoStack instance:

   {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Principal": {
             "Federated": "arn:aws:iam::${<aws_account_id>}:oidc-provider/${<oidc_provider>}" 1
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
             "StringEquals": {
               "${OIDC_PROVIDER}:sub": [
                 "system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}" 2
                 "system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}-query-frontend"
              ]
            }
          }
        }
       ]
   }

   1

   OIDC provider that you have configured on the OpenShift Container Platform. You can get the configured OIDC provider value also by running the following command: $ oc get authentication cluster -o json | jq -r '.spec.serviceAccountIssuer' | sed 'shttp[s]*://~g'.

   2

   Namespace in which you intend to create the TempoStack instance.

3. Create an AWS IAM role by attaching the trust.json policy file that you created:

   $ aws iam create-role \
         --role-name "tempo-s3-access" \
         --assume-role-policy-document "file:///tmp/trust.json" \
         --query Role.Arn \
         --output text

4. Attach an AWS IAM policy to the created role:

   $ aws iam attach-role-policy \
         --role-name "tempo-s3-access" \
         --policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess"

5. In the OpenShift Container Platform, create an object storage secret with keys as follows:

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     bucket: <s3_bucket_name>
     region: <s3_region>
     role_arn: <s3_role_arn>
   type: Opaque

Procedure

1. To enable metrics of a TempoStack instance, set the spec.observability.metrics.createServiceMonitors field to true:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: <name>
   spec:
     observability:
       metrics:
         createServiceMonitors: true

2. To enable alerts for a TempoStack instance, set the spec.observability.metrics.createPrometheusRules field to true:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: <name>
   spec:
     observability:
       metrics:
         createPrometheusRules: true

1. Go to Observe Targets, filter for Source: User, and check that ServiceMonitors in the format tempo-<instance_name>-<component> have the Up status.
2. To verify that alerts are set up correctly, go to Observe Alerting Alerting rules, filter for Source: User, and check that the Alert rules for the TempoStack instance components are available.

1. Go to Observe Targets, filter for Source: Platform, and search for tempo-operator, which must have the Up status.
2. To verify that alerts are set up correctly, go to Observe Alerting Alerting rules, filter for Source: Platform, and locate the Alert rules for the Tempo Operator.