Chapter 4. Distributed tracing platform (Tempo)

Procedure

1. Install the Tempo Operator:

   1. Go to Operators OperatorHub and search for Tempo Operator.

   2. Select the Tempo Operator that is OpenShift Operator for Tempo Install Install View Operator.

      Important

      This installs the Operator with the default presets:

      • Update channel stable
      • Installation mode All namespaces on the cluster
      • Installed Namespace openshift-tempo-operator
      • Update approval Automatic
   3. In the Details tab of the page of the installed Operator, under ClusterServiceVersion details, verify that the installation Status is Succeeded.
2. Create a project of your choice for the TempoStack instance that you will create in a subsequent step: go to Home Projects Create Project.

3. In the project that you created for the TempoStack instance, create a secret for your object storage bucket: go to Workloads Secrets Create From YAML.

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

4. Create a TempoStack instance.

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Go to Operators Installed Operators.
   2. Select TempoStack Create TempoStack YAML view.

   3. In the YAML view, customize the TempoStack custom resource (CR):

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: sample
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: 1Gi
        storage:
          secret:
            name: <secret-name> 1
            type: <secret-provider> 2
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      1

      The value of the name in the metadata of the secret.

      2

      The accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation.

      Example of a TempoStack CR for AWS S3 and MinIO storage

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: simplest
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: 1Gi
        storage:
          secret:
            name: minio-test
            type: s3
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the TempoStack instance.
2. Go to Operators Installed Operators to verify that the Status of the TempoStack instance is Condition: Ready.
3. Go to Workloads Pods to verify that all the component pods of the TempoStack instance are running.

4. Access the Tempo console:

   1. Go to Networking Routes and Ctrl+F to search for tempo.
   2. In the Location column, open the URL to access the Tempo console.

   3. Select Log In With OpenShift to use your cluster administrator credentials for the web console.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Install the Tempo Operator:

   1. Create a project for the Tempo Operator by running the following command:

      $ oc apply -f - << EOF
      apiVersion: project.openshift.io/v1
      kind: Project
      metadata:
        labels:
          kubernetes.io/metadata.name: openshift-tempo-operator
          openshift.io/cluster-monitoring: "true"
        name: openshift-tempo-operator
      EOF

   2. Create an Operator group by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-tempo-operator
        namespace: openshift-tempo-operator
      spec:
        upgradeStrategy: Default
      EOF

   3. Create a subscription by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: tempo-product
        namespace: openshift-tempo-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: tempo-product
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

   4. Check the operator status by running the following command:

      $ oc get csv -n openshift-tempo-operator

2. Run the following command to create a project of your choice for the TempoStack instance that you will create in a subsequent step:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: <project_of_tempostack_instance>
   EOF

3. In the project that you created for the TempoStack instance, create a secret for your object storage bucket by running the following command:

   $ oc apply -f - << EOF
   <object_storage_secret>
   EOF

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

4. Create a TempoStack instance in the project that you created for the TempoStack instance.

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Customize the TempoStack custom resource (CR):

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: sample
        namespace: <project_of_tempostack_instance>
      spec:
        storageSize: 1Gi
        storage:
            secret:
                name: <secret-name> 1
                type: <secret-provider> 2
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      1

      The value of the name in the metadata of the secret.

      2

      The accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation.

      Example of a TempoStack CR for AWS S3 and MinIO storage

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: simplest
        namespace: project_of_tempostack_instance
      spec:
        storageSize: 1Gi
        storage:
          secret:
            name: minio-test
            type: s3
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true
              ingress:
                route:
                  termination: edge
                type: route

      The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.

   2. Apply the customized CR by running the following command.

      $ oc apply -f - << EOF
      <TempoStack_custom_resource>
      EOF

Verification

1. Verify that the status of all TempoStack components is Running and the conditions are type: Ready by running the following command:

   $ oc get tempostacks.tempo.grafana.com simplest -o yaml

2. Verify that all the TempoStack component pods are running by running the following command:

   $ oc get pods

3. Access the Tempo console:

   1. Query the route details by running the following command:

      $ export TEMPO_URL=$(oc get route -n <control_plane_namespace> tempo -o jsonpath='{.spec.host}')

   2. Open https://<route_from_previous_step> in a web browser.

   3. Log in using your cluster administrator credentials for the web console.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Go to Operators Installed Operators Tempo Operator TempoStack.
2. To remove the TempoStack instance, select Delete TempoStack Delete.
3. Optional: Remove the Tempo Operator.

Procedure

1. Get the name of the TempoStack instance by running the following command:

   $ oc get deployments -n <project_of_tempostack_instance>

2. Remove the TempoStack instance by running the following command:

   $ oc delete tempo <tempostack_instance_name> -n <project_of_tempostack_instance>

3. Optional: Remove the Tempo Operator.

Verification

1. Run the following command to verify that the TempoStack instance is not found in the output, which indicates its successful removal:

   $ oc get deployments -n <project_of_tempostack_instance>