Chapter 3. Distributed tracing platform (Tempo)

Procedure

1. Go to Operators OperatorHub and search for Tempo Operator.

2. Select the Tempo Operator that is provided by Red Hat.

   Important

   The following selections are the default presets for this Operator:

   • Update channel stable
   • Installation mode All namespaces on the cluster
   • Installed Namespace openshift-tempo-operator
   • Update approval Automatic
3. Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
4. Select Install Install View Operator.

Procedure

1. Create a project for the Tempo Operator by running the following command:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     labels:
       kubernetes.io/metadata.name: openshift-tempo-operator
       openshift.io/cluster-monitoring: "true"
     name: openshift-tempo-operator
   EOF

2. Create an Operator group by running the following command:

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-tempo-operator
     namespace: openshift-tempo-operator
   spec:
     upgradeStrategy: Default
   EOF

3. Create a subscription by running the following command:

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: tempo-product
     namespace: openshift-tempo-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: tempo-product
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

Procedure

1. Create an AWS S3 bucket.

2. Create the following trust.json file for the AWS IAM policy that will set up a trust relationship for the AWS IAM role, created in the next step, with the service account of the TempoStack instance:

   {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Principal": {
             "Federated": "arn:aws:iam::${<aws_account_id>}:oidc-provider/${<oidc_provider>}" 1
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
             "StringEquals": {
               "${OIDC_PROVIDER}:sub": [
                 "system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}" 2
                 "system:serviceaccount:${<openshift_project_for_tempostack>}:tempo-${<tempostack_cr_name>}-query-frontend"
              ]
            }
          }
        }
       ]
   }

   1

   OIDC provider that you have configured on the OpenShift Container Platform. You can get the configured OIDC provider value also by running the following command: $ oc get authentication cluster -o json | jq -r '.spec.serviceAccountIssuer' | sed 'shttp[s]*://~g'.

   2

   Namespace in which you intend to create the TempoStack instance.

3. Create an AWS IAM role by attaching the trust.json policy file that you created:

   $ aws iam create-role \
         --role-name "tempo-s3-access" \
         --assume-role-policy-document "file:///tmp/trust.json" \
         --query Role.Arn \
         --output text

4. Attach an AWS IAM policy to the created role:

   $ aws iam attach-role-policy \
         --role-name "tempo-s3-access" \
         --policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess"

5. In the OpenShift Container Platform, create an object storage secret with keys as follows:

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     bucket: <s3_bucket_name>
     region: <s3_region>
     role_arn: <s3_role_arn>
   type: Opaque

Procedure

1. On IBM Cloud, create an object store bucket.

2. On IBM Cloud, create a service key for connecting to the object store bucket by running the following command:

   $ ibmcloud resource service-key-create <tempo_bucket> Writer \
     --instance-name <tempo_bucket> --parameters '{"HMAC":true}'

3. On IBM Cloud, create a secret with the bucket credentials by running the following command:

   $ oc -n <namespace> create secret generic <ibm_cos_secret> \
     --from-literal=bucket="<tempo_bucket>" \
     --from-literal=endpoint="<ibm_bucket_endpoint>" \
     --from-literal=access_key_id="<ibm_bucket_access_key>" \
     --from-literal=access_key_secret="<ibm_bucket_secret_key>"

4. On OpenShift Container Platform, create an object storage secret with keys as follows:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <ibm_cos_secret>
   stringData:
     bucket: <tempo_bucket>
     endpoint: <ibm_bucket_endpoint>
     access_key_id: <ibm_bucket_access_key>
     access_key_secret: <ibm_bucket_secret_key>
   type: Opaque

5. On OpenShift Container Platform, set the storage section in the TempoStack custom resource as follows:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   # ...
   spec:
   # ...
     storage:
       secret:
         name: <ibm_cos_secret> 1
         type: s3
   # ...

   1

   Name of the secret that contains the IBM Cloud Storage access and secret keys.

Procedure

1. Define the tenants by adding the tenantName and tenantId parameters with your values of choice to the TempoStack custom resource (CR):

   Tenant example in a TempoStack CR

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: redmetrics
   spec:
   # ...
     tenants:
       mode: openshift
       authentication:
         - tenantName: dev 1
           tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 2
   # ...

   1

   A tenantName value of the user’s choice.

   2

   A tenantId value of the user’s choice.

2. Add the tenants to a cluster role with the read (get) permissions to read traces.

   Example RBAC configuration in a ClusterRole resource

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: tempostack-traces-reader
   rules:
     - apiGroups:
         - 'tempo.grafana.com'
       resources: 1
         - dev
         - prod
       resourceNames:
         - traces
       verbs:
         - 'get' 2

   1

   Lists the tenants, dev and prod in this example, which are defined by using the tenantName parameter in the previous step.

   2

   Enables the read operation for the listed tenants.

3. Grant authenticated users the read permissions for trace data by defining a cluster role binding for the cluster role from the previous step.

   Example RBAC configuration in a ClusterRoleBinding resource

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: tempostack-traces-reader
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: tempostack-traces-reader
   subjects:
     - kind: Group
       apiGroup: rbac.authorization.k8s.io
       name: system:authenticated 1

   1

   Grants all authenticated users the read permissions for trace data.

Procedure

1. Create a service account for use with OpenTelemetry Collector.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector
     namespace: <project_of_opentelemetry_collector_instance>

2. Add the tenants to a cluster role with the write (create) permissions to write traces.

   Example RBAC configuration in a ClusterRole resource

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: tempostack-traces-write
   rules:
     - apiGroups:
         - 'tempo.grafana.com'
       resources: 1
         - dev
       resourceNames:
         - traces
       verbs:
         - 'create' 2

   1

   Lists the tenants.

   2

   Enables the write operation.

3. Grant the OpenTelemetry Collector the write permissions by defining a cluster role binding to attach the OpenTelemetry Collector service account.

   Example RBAC configuration in a ClusterRoleBinding resource

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: tempostack-traces
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: tempostack-traces-write
   subjects:
     - kind: ServiceAccount
       name: otel-collector 1
       namespace: otel

   1

   The service account that you created in a previous step. The client uses it when exporting trace data.

4. Configure the OpenTelemetryCollector custom resource as follows:

   • Add the bearertokenauth extension and a valid token to the tracing pipeline service.
   • Add the tenant name in the otlp/otlphttp exporters as the X-Scope-OrgID headers.

   • Enable TLS with a valid certificate authority file.

     Sample OpenTelemetry CR configuration

     apiVersion: opentelemetry.io/v1beta1
     kind: OpenTelemetryCollector
     metadata:
       name: cluster-collector
       namespace: <project_of_tempostack_instance>
     spec:
       mode: deployment
       serviceAccount: otel-collector 1
       config: |
           extensions:
             bearertokenauth: 2
               filename: "/var/run/secrets/kubernetes.io/serviceaccount/token" 3
           exporters:
             otlp/dev: 4
               endpoint: sample-gateway.tempo.svc.cluster.local:8090
               tls:
                 insecure: false
                 ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt" 5
               auth:
                 authenticator: bearertokenauth
               headers:
                 X-Scope-OrgID: "dev" 6
             otlphttp/dev: 7
               endpoint: https://sample-gateway.<project_of_tempostack_instance>.svc.cluster.local:8080/api/traces/v1/dev
               tls:
                 insecure: false
                 ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
               auth:
                 authenticator: bearertokenauth
               headers:
                 X-Scope-OrgID: "dev"
           service:
             extensions: [bearertokenauth]
             pipelines:
               traces:
                 exporters: [otlp/dev] 8
     
     # ...

     1

     Service account configured with write permissions.

     2

     Bearer Token extension to use service account token.

     3

     The service account token. The client sends the token to the tracing pipeline service as the bearer token header.

     4

     Specify either the OTLP gRPC Exporter (otlp/dev) or the OTLP HTTP Exporter (otlphttp/dev).

     5

     Enabled TLS with a valid service CA file.

     6

     Header with tenant name.

     7

     Specify either the OTLP gRPC Exporter (otlp/dev) or the OTLP HTTP Exporter (otlphttp/dev).

     8

     The exporter you specified in exporters section of the CR.

Procedure

1. Go to Home Projects Create Project to create a project of your choice for the TempoStack instance that you will create in a subsequent step.

2. Go to Workloads Secrets Create From YAML to create a secret for your object storage bucket in the project that you created for the TempoStack instance. For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoStack instance.

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Go to Operators Installed Operators.
   2. Select TempoStack Create TempoStack YAML view.

   3. In the YAML view, customize the TempoStack custom resource (CR):

      Example TempoStack CR for AWS S3 and MinIO storage and two tenants

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack 1
      metadata:
        name: simplest
        namespace: <project_of_tempostack_instance> 2
      spec:
        storage: 3
          secret: 4
            name: <secret_name> 5
            type: <secret_provider> 6
        storageSize: <value>Gi 7
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        tenants:
          mode: openshift 8
          authentication: 9
            - tenantName: dev 10
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 11
            - tenantName: prod
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
        template:
          gateway:
            enabled: true 12
          queryFrontend:
            jaegerQuery:
              enabled: true 13

      1

      This CR creates a TempoStack deployment, which is configured to receive Jaeger Thrift over the HTTP and OpenTelemetry Protocol (OTLP).`

      2

      The namespace that you have chosen for the TempoStack deployment.

      3

      Specifies the storage for storing traces.

      4

      The secret you created in step 2 for the object storage that had been set up as one of the prerequisites.

      5

      The value of the name field in the metadata section of the secret. For example: minio.

      6

      The accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation. For example: s3.

      7

      The size of the persistent volume claim for the Tempo Write-Ahead Logging (WAL). The default is 10Gi. For example: 1Gi.

      8

      The value must be openshift.

      9

      The list of tenants.

      10

      The tenant name, which is to be provided in the X-Scope-OrgId header when ingesting the data.

      11

      The unique identifier of the tenant. Must be unique throughout the lifecycle of the TempoStack deployment. The distributed tracing platform (Tempo) uses this ID to prefix objects in the object storage. You can reuse the value of the UUID or tempoName field.

      12

      Enables a gateway that performs authentication and authorization. The Jaeger UI is exposed at http://<gateway_ingress>/api/traces/v1/<tenant_name>/search.

      13

      Exposes the Jaeger UI, which visualizes the data, via a route.

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the TempoStack instance.
2. Go to Operators Installed Operators to verify that the Status of the TempoStack instance is Condition: Ready.
3. Go to Workloads Pods to verify that all the component pods of the TempoStack instance are running.

4. Access the Tempo console:

   1. Go to Networking Routes and Ctrl+F to search for tempo.

   2. In the Location column, open the URL to access the Tempo console.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Run the following command to create a project of your choice for the TempoStack instance that you will create in a subsequent step:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: <project_of_tempostack_instance>
   EOF

2. In the project that you created for the TempoStack instance, create a secret for your object storage bucket by running the following command:

   $ oc apply -f - << EOF
   <object_storage_secret>
   EOF

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoStack instance in the project that you created for it:

   Note

   You can create multiple TempoStack instances in separate projects on the same cluster.

   1. Customize the TempoStack custom resource (CR):

      Example TempoStack CR for AWS S3 and MinIO storage and two tenants

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack 1
      metadata:
        name: simplest
        namespace: <project_of_tempostack_instance> 2
      spec:
        storage: 3
          secret: 4
            name: <secret_name> 5
            type: <secret_provider> 6
        storageSize: <value>Gi 7
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        tenants:
          mode: openshift 8
          authentication: 9
            - tenantName: dev 10
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 11
            - tenantName: prod
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
        template:
          gateway:
            enabled: true 12
          queryFrontend:
            jaegerQuery:
              enabled: true 13

      1

      This CR creates a TempoStack deployment, which is configured to receive Jaeger Thrift over the HTTP and OpenTelemetry Protocol (OTLP).`

      2

      The namespace that you have chosen for the TempoStack deployment.

      3

      Specifies the storage for storing traces.

      4

      The secret you created in step 2 for the object storage that had been set up as one of the prerequisites.

      5

      The value of the name field in the metadata section of the secret. For example: minio.

      6

      The accepted values are azure for Azure Blob Storage; gcs for Google Cloud Storage; and s3 for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation. For example: s3.

      7

      The size of the persistent volume claim for the Tempo Write-Ahead Logging (WAL). The default is 10Gi. For example: 1Gi.

      8

      The value must be openshift.

      9

      The list of tenants.

      10

      The tenant name, which is to be provided in the X-Scope-OrgId header when ingesting the data.

      11

      The unique identifier of the tenant. Must be unique throughout the lifecycle of the TempoStack deployment. The distributed tracing platform (Tempo) uses this ID to prefix objects in the object storage. You can reuse the value of the UUID or tempoName field.

      12

      Enables a gateway that performs authentication and authorization. The Jaeger UI is exposed at http://<gateway_ingress>/api/traces/v1/<tenant_name>/search.

      13

      Exposes the Jaeger UI, which visualizes the data, via a route.

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <tempostack_cr>
      EOF

Verification

1. Verify that the status of all TempoStack components is Running and the conditions are type: Ready by running the following command:

   $ oc get tempostacks.tempo.grafana.com simplest -o yaml

2. Verify that all the TempoStack component pods are running by running the following command:

   $ oc get pods

3. Access the Tempo console:

   1. Query the route details by running the following command:

      $ oc get route

   2. Open https://<route_from_previous_step> in a web browser.

      Note

      The Tempo console initially shows no trace data following the Tempo console installation.

Procedure

1. Go to Home Projects Create Project to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step.

2. Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.

   Important

   Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.

   Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this in Workloads Secrets Create From YAML.

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoMonolithic instance:

   Note

   You can create multiple TempoMonolithic instances in separate projects on the same cluster.

   1. Go to Operators Installed Operators.
   2. Select TempoMonolithic Create TempoMonolithic YAML view.

   3. In the YAML view, customize the TempoMonolithic custom resource (CR).

      Example TempoMonolithic CR

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoMonolithic 1
      metadata:
        name: <metadata_name>
        namespace: <project_of_tempomonolithic_instance> 2
      spec:
        storage: 3
          traces:
            backend: <supported_storage_type> 4
            size: <value>Gi 5
            s3: 6
              secret: <secret_name> 7
          tls: 8
            enabled: true
            caName: <ca_certificate_configmap_name> 9
        jaegerui:
          enabled: true 10
          route:
            enabled: true 11
        resources: 12
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m
        multitenancy:
          enabled: true
          mode: openshift
          authentication: 13
            - tenantName: dev 14
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 15
            - tenantName: prod
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"

      1

      This CR creates a TempoMonolithic deployment with trace ingestion in the OTLP protocol.

      2

      The namespace that you have chosen for the TempoMonolithic deployment.

      3

      Specifies the storage for storing traces.

      4

      Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv. The accepted values for object storage are s3, gcs, or azure, depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down.

      5

      Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi. For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi. For object storage, this means the size of the persistent volume claim for the Tempo Write-Ahead Logging (WAL), where the default is 10Gi.

      6

      Optional: For object storage, the type of object storage. The accepted values are s3, gcs, and azure, depending on the used object store type.

      7

      Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup".

      8

      Optional.

      9

      Optional: Name of a ConfigMap object that contains a CA certificate.

      10

      Exposes the Jaeger UI, which visualizes the data, via a route.

      11

      Enables creation of a route for the Jaeger UI.

      12

      Optional.

      13

      Lists the tenants.

      14

      The tenant name from the X-Scope-OrgId header when ingesting the data.

      15

      The unique identifier of the tenant. Must be unique throughout the lifecycle of the TempoMonolithic deployment. This ID will be added as a prefix to the objects in the object storage. You can reuse the value of the UUID or tempoName field.

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the TempoMonolithic instance.
2. Go to Operators Installed Operators to verify that the Status of the TempoMonolithic instance is Condition: Ready.
3. Go to Workloads Pods to verify that the pod of the TempoMonolithic instance is running.

4. Access the Jaeger UI:

   1. Go to Networking Routes and Ctrl+F to search for jaegerui.

      Note

      The Jaeger UI uses the tempo-<metadata_name_of_TempoMonolithic_CR>-jaegerui route.

   2. In the Location column, open the URL to access the Jaeger UI.

5. When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_TempoMonolithic_CR>:4317 (OTLP/gRPC) and tempo-<metadata_name_of_TempoMonolithic_CR>:4318 (OTLP/HTTP) endpoints inside the cluster.

   The Tempo API is available at the tempo-<metadata_name_of_TempoMonolithic_CR>:3200 endpoint inside the cluster.

Procedure

1. Run the following command to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step:

   $ oc apply -f - << EOF
   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: <project_of_tempomonolithic_instance>
   EOF

2. Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.

   Important

   Object storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.

   Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this by running the following command:

   $ oc apply -f - << EOF
   <object_storage_secret>
   EOF

   For more information, see "Object storage setup".

   Example secret for Amazon S3 and MinIO storage

   apiVersion: v1
   kind: Secret
   metadata:
     name: minio-test
   stringData:
     endpoint: http://minio.minio.svc:9000
     bucket: tempo
     access_key_id: tempo
     access_key_secret: <secret>
   type: Opaque

3. Create a TempoMonolithic instance in the project that you created for it.

   Tip

   You can create multiple TempoMonolithic instances in separate projects on the same cluster.

   1. Customize the TempoMonolithic custom resource (CR).

      Example TempoMonolithic CR

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoMonolithic 1
      metadata:
        name: <metadata_name>
        namespace: <project_of_tempomonolithic_instance> 2
      spec:
        storage: 3
          traces:
            backend: <supported_storage_type> 4
            size: <value>Gi 5
            s3: 6
              secret: <secret_name> 7
          tls: 8
            enabled: true
            caName: <ca_certificate_configmap_name> 9
        jaegerui:
          enabled: true 10
          route:
            enabled: true 11
        resources: 12
          total:
            limits:
              memory: <value>Gi
              cpu: <value>m
        multitenancy:
          enabled: true
          mode: openshift
          authentication: 13
            - tenantName: dev 14
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 15
            - tenantName: prod
              tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"

      1

      This CR creates a TempoMonolithic deployment with trace ingestion in the OTLP protocol.

      2

      The namespace that you have chosen for the TempoMonolithic deployment.

      3

      Specifies the storage for storing traces.

      4

      Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for a persistent volume is pv. The accepted values for object storage are s3, gcs, or azure, depending on the used object store type. The default value is memory for the tmpfs in-memory storage, which is only appropriate for development, testing, demonstrations, and proof-of-concept environments because the data does not persist when the pod is shut down.

      5

      Memory size: For in-memory storage, this means the size of the tmpfs volume, where the default is 2Gi. For a persistent volume, this means the size of the persistent volume claim, where the default is 10Gi. For object storage, this means the size of the persistent volume claim for the Tempo Write-Ahead Logging (WAL), where the default is 10Gi.

      6

      Optional: For object storage, the type of object storage. The accepted values are s3, gcs, and azure, depending on the used object store type.

      7

      Optional: For object storage, the value of the name in the metadata of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup".

      8

      Optional.

      9

      Optional: Name of a ConfigMap object that contains a CA certificate.

      10

      Exposes the Jaeger UI, which visualizes the data, via a route.

      11

      Enables creation of a route for the Jaeger UI.

      12

      Optional.

      13

      Lists the tenants.

      14

      The tenant name from the X-Scope-OrgId header when ingesting the data.

      15

      The unique identifier of the tenant. Must be unique throughout the lifecycle of the TempoMonolithic deployment. This ID will be added as a prefix to the objects in the object storage. You can reuse the value of the UUID or tempoName field.

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <tempomonolithic_cr>
      EOF

Verification

1. Verify that the status of all TempoMonolithic components is Running and the conditions are type: Ready by running the following command:

   $ oc get tempomonolithic.tempo.grafana.com <metadata_name_of_tempomonolithic_cr> -o yaml

2. Run the following command to verify that the pod of the TempoMonolithic instance is running:

   $ oc get pods

3. Access the Jaeger UI:

   1. Query the route details for the tempo-<metadata_name_of_tempomonolithic_cr>-jaegerui route by running the following command:

      $ oc get route

   2. Open https://<route_from_previous_step> in a web browser.

4. When the pod of the TempoMonolithic instance is ready, you can send traces to the tempo-<metadata_name_of_tempomonolithic_cr>:4317 (OTLP/gRPC) and tempo-<metadata_name_of_tempomonolithic_cr>:4318 (OTLP/HTTP) endpoints inside the cluster.

   The Tempo API is available at the tempo-<metadata_name_of_tempomonolithic_cr>:3200 endpoint inside the cluster.

Procedure

1. To enable metrics of a TempoStack instance, set the spec.observability.metrics.createServiceMonitors field to true:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: <name>
   spec:
     observability:
       metrics:
         createServiceMonitors: true

2. To enable alerts for a TempoStack instance, set the spec.observability.metrics.createPrometheusRules field to true:

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: <name>
   spec:
     observability:
       metrics:
         createPrometheusRules: true

1. Go to Observe Targets, filter for Source: User, and check that ServiceMonitors in the format tempo-<instance_name>-<component> have the Up status.
2. To verify that alerts are set up correctly, go to Observe Alerting Alerting rules, filter for Source: User, and check that the Alert rules for the TempoStack instance components are available.

1. Go to Observe Targets, filter for Source: Platform, and search for tempo-operator, which must have the Up status.
2. To verify that alerts are set up correctly, go to Observe Alerting Alerting rules, filter for Source: Platform, and locate the Alert rules for the Tempo Operator.