Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 3. Distributed tracing platform (Tempo)
3.1. Installing
Installing the distributed tracing platform (Tempo) requires the Tempo Operator and choosing which type of deployment is best for your use case:
- For microservices mode, deploy a TempoStack instance in a dedicated OpenShift project.
- For monolithic mode, deploy a TempoMonolithic instance in a dedicated OpenShift project.
Using object storage requires setting up a supported object store and creating a secret for the object store credentials before deploying a TempoStack or TempoMonolithic instance.
3.1.1. Installing the Tempo Operator
You can install the Tempo Operator by using the web console or the command line.
3.1.1.1. Installing the Tempo Operator by using the web console
You can install the Tempo Operator from the Administrator view of the web console.
Prerequisites
-
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the
cluster-admin
role. -
For Red Hat OpenShift Dedicated, you must be logged in using an account with the
dedicated-admin
role. You have completed setting up the required object storage by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
WarningObject storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo).
Procedure
-
Go to Operators
OperatorHub and search for Tempo Operator
. Select the Tempo Operator that is provided by Red Hat.
ImportantThe following selections are the default presets for this Operator:
-
Update channel
stable -
Installation mode
All namespaces on the cluster -
Installed Namespace
openshift-tempo-operator -
Update approval
Automatic
-
Update channel
- Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
-
Select Install
Install View Operator.
Verification
- In the Details tab of the page of the installed Operator, under ClusterServiceVersion details, verify that the installation Status is Succeeded.
3.1.1.2. Installing the Tempo Operator by using the CLI
You can install the Tempo Operator from the command line.
Prerequisites
An active OpenShift CLI (
oc
) session by a cluster administrator with thecluster-admin
role.Tip-
Ensure that your OpenShift CLI (
oc
) version is up to date and matches your OpenShift Container Platform version. Run
oc login
:$ oc login --username=<your_username>
-
Ensure that your OpenShift CLI (
You have completed setting up the required object storage by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
WarningObject storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo).
Procedure
Create a project for the Tempo Operator by running the following command:
$ oc apply -f - << EOF apiVersion: project.openshift.io/v1 kind: Project metadata: labels: kubernetes.io/metadata.name: openshift-tempo-operator openshift.io/cluster-monitoring: "true" name: openshift-tempo-operator EOF
Create an Operator group by running the following command:
$ oc apply -f - << EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-tempo-operator namespace: openshift-tempo-operator spec: upgradeStrategy: Default EOF
Create a subscription by running the following command:
$ oc apply -f - << EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: tempo-product namespace: openshift-tempo-operator spec: channel: stable installPlanApproval: Automatic name: tempo-product source: redhat-operators sourceNamespace: openshift-marketplace EOF
Verification
Check the Operator status by running the following command:
$ oc get csv -n openshift-tempo-operator
3.1.2. Installing a TempoStack instance
You can install a TempoStack instance by using the web console or the command line.
3.1.2.1. Installing a TempoStack instance by using the web console
You can install a TempoStack instance from the Administrator view of the web console.
Prerequisites
-
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the
cluster-admin
role. -
For Red Hat OpenShift Dedicated, you must be logged in using an account with the
dedicated-admin
role. You have completed setting up the required object storage by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
WarningObject storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo).
Procedure
-
Go to Home
Projects Create Project to create a project of your choice for the TempoStack instance that you will create in a subsequent step. Go to Workloads
Secrets Create From YAML to create a secret for your object storage bucket in the project that you created for the TempoStack instance. For more information, see "Object storage setup". Example secret for Amazon S3 and MinIO storage
apiVersion: v1 kind: Secret metadata: name: minio-test stringData: endpoint: http://minio.minio.svc:9000 bucket: tempo access_key_id: tempo access_key_secret: <secret> type: Opaque
Create a TempoStack instance.
NoteYou can create multiple TempoStack instances in separate projects on the same cluster.
-
Go to Operators
Installed Operators. -
Select TempoStack
Create TempoStack YAML view. In the YAML view, customize the
TempoStack
custom resource (CR):apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: sample namespace: <project_of_tempostack_instance> spec: storageSize: 1Gi storage: secret: 1 name: <secret_name> 2 type: <secret_provider> 3 template: queryFrontend: jaegerQuery: enabled: true ingress: route: termination: edge type: route
- 1
- The secret you created in step 2 for the object storage that had been set up as one of the prerequisites.
- 2
- The value of the
name
in themetadata
of the secret. - 3
- The accepted values are
azure
for Azure Blob Storage;gcs
for Google Cloud Storage; ands3
for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation.
Example of a
TempoStack
CR for AWS S3 and MinIO storageapiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: simplest namespace: <project_of_tempostack_instance> spec: storageSize: 1Gi storage: 1 secret: name: minio-test type: s3 resources: total: limits: memory: 2Gi cpu: 2000m template: queryFrontend: jaegerQuery: 2 enabled: true ingress: route: termination: edge type: route
- 1
- In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2.
- 2
- The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.
- Select Create.
-
Go to Operators
Verification
- Use the Project: dropdown list to select the project of the TempoStack instance.
-
Go to Operators
Installed Operators to verify that the Status of the TempoStack instance is Condition: Ready. -
Go to Workloads
Pods to verify that all the component pods of the TempoStack instance are running. Access the Tempo console:
-
Go to Networking
Routes and Ctrl+F to search for tempo
. In the Location column, open the URL to access the Tempo console.
NoteThe Tempo console initially shows no trace data following the Tempo console installation.
-
Go to Networking
3.1.2.2. Installing a TempoStack instance by using the CLI
You can install a TempoStack instance from the command line.
Prerequisites
An active OpenShift CLI (
oc
) session by a cluster administrator with thecluster-admin
role.Tip-
Ensure that your OpenShift CLI (
oc
) version is up to date and matches your OpenShift Container Platform version. Run the
oc login
command:$ oc login --username=<your_username>
-
Ensure that your OpenShift CLI (
You have completed setting up the required object storage by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, Google Cloud Storage. For more information, see "Object storage setup".
WarningObject storage is required and not included with the distributed tracing platform (Tempo). You must choose and set up object storage by a supported provider before installing the distributed tracing platform (Tempo).
Procedure
Run the following command to create a project of your choice for the TempoStack instance that you will create in a subsequent step:
$ oc apply -f - << EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: <project_of_tempostack_instance> EOF
In the project that you created for the TempoStack instance, create a secret for your object storage bucket by running the following command:
$ oc apply -f - << EOF <object_storage_secret> EOF
For more information, see "Object storage setup".
Example secret for Amazon S3 and MinIO storage
apiVersion: v1 kind: Secret metadata: name: minio-test stringData: endpoint: http://minio.minio.svc:9000 bucket: tempo access_key_id: tempo access_key_secret: <secret> type: Opaque
Create a TempoStack instance in the project that you created for it:
NoteYou can create multiple TempoStack instances in separate projects on the same cluster.
Customize the
TempoStack
custom resource (CR):apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: sample namespace: <project_of_tempostack_instance> spec: storageSize: 1Gi storage: secret: 1 name: <secret_name> 2 type: <secret_provider> 3 template: queryFrontend: jaegerQuery: enabled: true ingress: route: termination: edge type: route
- 1
- The secret you created in step 2 for the object storage that had been set up as one of the prerequisites.
- 2
- The value of the
name
in themetadata
of the secret. - 3
- The accepted values are
azure
for Azure Blob Storage;gcs
for Google Cloud Storage; ands3
for Amazon S3, MinIO, or Red Hat OpenShift Data Foundation.
Example of a
TempoStack
CR for AWS S3 and MinIO storageapiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: simplest namespace: <project_of_tempostack_instance> spec: storageSize: 1Gi storage: 1 secret: name: minio-test type: s3 resources: total: limits: memory: 2Gi cpu: 2000m template: queryFrontend: jaegerQuery: 2 enabled: true ingress: route: termination: edge type: route
- 1
- In this example, the object storage was set up as one of the prerequisites, and the object storage secret was created in step 2.
- 2
- The stack deployed in this example is configured to receive Jaeger Thrift over HTTP and OpenTelemetry Protocol (OTLP), which permits visualizing the data with the Jaeger UI.
Apply the customized CR by running the following command:
$ oc apply -f - << EOF <tempostack_cr> EOF
Verification
Verify that the
status
of all TempoStackcomponents
isRunning
and theconditions
aretype: Ready
by running the following command:$ oc get tempostacks.tempo.grafana.com simplest -o yaml
Verify that all the TempoStack component pods are running by running the following command:
$ oc get pods
Access the Tempo console:
Query the route details by running the following command:
$ oc get route
Open
https://<route_from_previous_step>
in a web browser.NoteThe Tempo console initially shows no trace data following the Tempo console installation.
3.1.3. Installing a TempoMonolithic instance
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can install a TempoMonolithic instance by using the web console or the command line.
The TempoMonolithic
custom resource (CR) creates a Tempo deployment in monolithic mode. All components of the Tempo deployment, such as the compactor, distributor, ingester, querier, and query frontend, are contained in a single container.
A TempoMonolithic instance supports storing traces in in-memory storage, a persistent volume, or object storage.
Tempo deployment in monolithic mode is preferred for a small deployment, demonstration, testing, and as a migration path of the Red Hat OpenShift distributed tracing platform (Jaeger) all-in-one deployment.
The monolithic deployment of Tempo does not scale horizontally. If you require horizontal scaling, use the TempoStack
CR for a Tempo deployment in microservices mode.
3.1.3.1. Installing a TempoMonolithic instance by using the web console
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can install a TempoMonolithic instance from the Administrator view of the web console.
Prerequisites
-
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the
cluster-admin
role. -
For Red Hat OpenShift Dedicated, you must be logged in using an account with the
dedicated-admin
role.
Procedure
-
Go to Home
Projects Create Project to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step. Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.
ImportantObject storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.
Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this in Workloads
Secrets Create From YAML. For more information, see "Object storage setup".
Example secret for Amazon S3 and MinIO storage
apiVersion: v1 kind: Secret metadata: name: minio-test stringData: endpoint: http://minio.minio.svc:9000 bucket: tempo access_key_id: tempo access_key_secret: <secret> type: Opaque
Create a TempoMonolithic instance:
NoteYou can create multiple TempoMonolithic instances in separate projects on the same cluster.
-
Go to Operators
Installed Operators. -
Select TempoMonolithic
Create TempoMonolithic YAML view. In the YAML view, customize the
TempoMonolithic
custom resource (CR).The following
TempoMonolithic
CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:apiVersion: tempo.grafana.com/v1alpha1 kind: TempoMonolithic metadata: name: <metadata_name> namespace: <project_of_tempomonolithic_instance> spec: storage: traces: backend: <supported_storage_type> 1 size: <value>Gi 2 s3: 3 secret: <secret_name> 4 jaegerui: enabled: true 5 route: enabled: true 6
- 1
- Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for the
tmpfs
in-memory storage ismemory
. The value for a persistent volume ispv
. The accepted values for object storage ares3
,gcs
, orazure
, depending on the used object store type. - 2
- Memory size: For in-memory storage, this means the size of the
tmpfs
volume, where the default is2Gi
. For a persistent volume, this means the size of the persistent volume claim, where the default is10Gi
. For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is10Gi
. - 3
- Optional: For object storage, the type of object storage. The accepted values are
s3
,gcs
, andazure
, depending on the used object store type. - 4
- Optional: For object storage, the value of the
name
in themetadata
of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup". - 5
- Enables the Jaeger UI.
- 6
- Enables creation of a route for the Jaeger UI.
- Select Create.
-
Go to Operators
Verification
- Use the Project: dropdown list to select the project of the TempoMonolithic instance.
-
Go to Operators
Installed Operators to verify that the Status of the TempoMonolithic instance is Condition: Ready. -
Go to Workloads
Pods to verify that the pod of the TempoMonolithic instance is running. Access the Jaeger UI:
Go to Networking
Routes and Ctrl+F to search for jaegerui
.NoteThe Jaeger UI uses the
tempo-<metadata_name_of_TempoMonolithic_CR>-jaegerui
route.- In the Location column, open the URL to access the Jaeger UI.
When the pod of the TempoMonolithic instance is ready, you can send traces to the
tempo-<metadata_name_of_TempoMonolithic_CR>:4317
(OTLP/gRPC) andtempo-<metadata_name_of_TempoMonolithic_CR>:4318
(OTLP/HTTP) endpoints inside the cluster.The Tempo API is available at the
tempo-<metadata_name_of_TempoMonolithic_CR>:3200
endpoint inside the cluster.
3.1.3.2. Installing a TempoMonolithic instance by using the CLI
The TempoMonolithic instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can install a TempoMonolithic instance from the command line.
Prerequisites
An active OpenShift CLI (
oc
) session by a cluster administrator with thecluster-admin
role.Tip-
Ensure that your OpenShift CLI (
oc
) version is up to date and matches your OpenShift Container Platform version. Run the
oc login
command:$ oc login --username=<your_username>
-
Ensure that your OpenShift CLI (
Procedure
Run the following command to create a project of your choice for the TempoMonolithic instance that you will create in a subsequent step:
$ oc apply -f - << EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: <project_of_tempomonolithic_instance> EOF
Decide which type of supported storage to use for storing traces: in-memory storage, a persistent volume, or object storage.
ImportantObject storage is not included with the distributed tracing platform (Tempo) and requires setting up an object store by a supported provider: Red Hat OpenShift Data Foundation, MinIO, Amazon S3, Azure Blob Storage, or Google Cloud Storage.
Additionally, opting for object storage requires creating a secret for your object storage bucket in the project that you created for the TempoMonolithic instance. You can do this by running the following command:
$ oc apply -f - << EOF <object_storage_secret> EOF
For more information, see "Object storage setup".
Example secret for Amazon S3 and MinIO storage
apiVersion: v1 kind: Secret metadata: name: minio-test stringData: endpoint: http://minio.minio.svc:9000 bucket: tempo access_key_id: tempo access_key_secret: <secret> type: Opaque
Create a TempoMonolithic instance in the project that you created for it.
TipYou can create multiple TempoMonolithic instances in separate projects on the same cluster.
Customize the
TempoMonolithic
custom resource (CR).The following
TempoMonolithic
CR creates a TempoMonolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a supported type of storage and exposing Jaeger UI via a route:apiVersion: tempo.grafana.com/v1alpha1 kind: TempoMonolithic metadata: name: <metadata_name> namespace: <project_of_tempomonolithic_instance> spec: storage: traces: backend: <supported_storage_type> 1 size: <value>Gi 2 s3: 3 secret: <secret_name> 4 jaegerui: enabled: true 5 route: enabled: true 6
- 1
- Type of storage for storing traces: in-memory storage, a persistent volume, or object storage. The value for the
tmpfs
in-memory storage ismemory
. The value for a persistent volume ispv
. The accepted values for object storage ares3
,gcs
, orazure
, depending on the used object store type. - 2
- Memory size: For in-memory storage, this means the size of the
tmpfs
volume, where the default is2Gi
. For a persistent volume, this means the size of the persistent volume claim, where the default is10Gi
. For object storage, this means the size of the persistent volume claim for the Tempo WAL, where the default is10Gi
. - 3
- Optional: For object storage, the type of object storage. The accepted values are
s3
,gcs
, andazure
, depending on the used object store type. - 4
- Optional: For object storage, the value of the
name
in themetadata
of the storage secret. The storage secret must be in the same namespace as the TempoMonolithic instance and contain the fields specified in "Table 1. Required secret parameters" in the section "Object storage setup". - 5
- Enables the Jaeger UI.
- 6
- Enables creation of a route for the Jaeger UI.
Apply the customized CR by running the following command:
$ oc apply -f - << EOF <tempomonolithic_cr> EOF
Verification
Verify that the
status
of all TempoMonolithiccomponents
isRunning
and theconditions
aretype: Ready
by running the following command:$ oc get tempomonolithic.tempo.grafana.com <metadata_name_of_tempomonolithic_cr> -o yaml
Run the following command to verify that the pod of the TempoMonolithic instance is running:
$ oc get pods
Access the Jaeger UI:
Query the route details for the
tempo-<metadata_name_of_tempomonolithic_cr>-jaegerui
route by running the following command:$ oc get route
-
Open
https://<route_from_previous_step>
in a web browser.
When the pod of the TempoMonolithic instance is ready, you can send traces to the
tempo-<metadata_name_of_tempomonolithic_cr>:4317
(OTLP/gRPC) andtempo-<metadata_name_of_tempomonolithic_cr>:4318
(OTLP/HTTP) endpoints inside the cluster.The Tempo API is available at the
tempo-<metadata_name_of_tempomonolithic_cr>:3200
endpoint inside the cluster.
3.1.4. Object storage setup
You can use the following configuration parameters when setting up a supported object storage.
Storage provider |
---|
Secret parameters |
|
MinIO |
See MinIO Operator.
|
Amazon S3 |
|
Microsoft Azure Blob Storage |
|
Google Cloud Storage on Google Cloud Platform (GCP) |
|
3.1.5. Additional resources
3.2. Configuring
The Tempo Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing platform (Tempo) resources. You can install the default configuration or modify the file.
3.2.1. Customizing your deployment
For information about configuring the back-end storage, see Understanding persistent storage and the appropriate configuration topic for your chosen storage option.
3.2.1.1. Default configuration options
The TempoStack
custom resource (CR) defines the architecture and settings to be used when creating the distributed tracing platform (Tempo) resources. You can modify these parameters to customize your distributed tracing platform (Tempo) implementation to your business needs.
Example of a generic Tempo YAML file
apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: name spec: storage: {} resources: {} storageSize: 200M replicationFactor: 1 retention: {} template: distributor:{} ingester: {} compactor: {} querier: {} queryFrontend: {} gateway: {}
Parameter | Description | Values | Default value |
---|---|---|---|
| API version to use when creating the object. |
|
|
| Defines the kind of Kubernetes object to create. |
| |
|
Data that uniquely identifies the object, including a |
OpenShift Container Platform automatically generates the | |
| Name for the object. | Name of your TempoStack instance. |
|
| Specification for the object to be created. |
Contains all of the configuration parameters for your TempoStack instance. When a common definition for all Tempo components is required, it is defined under the | N/A |
| Resources assigned to the TempoStack instance. | ||
| Storage size for ingester PVCs. | ||
| Configuration for the replication factor. | ||
| Configuration options for retention of traces. | ||
|
Configuration options that define the storage. All storage-related options must be placed under | ||
|
Configuration options for the Tempo | ||
|
Configuration options for the Tempo | ||
|
Configuration options for the Tempo | ||
|
Configuration options for the Tempo | ||
|
Configuration options for the Tempo | ||
|
Configuration options for the Tempo |
Minimum required configuration
The following is the required minimum for creating a distributed tracing platform (Tempo) deployment with the default settings:
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simplest
spec:
storage: 1
secret:
name: minio
type: s3
resources:
total:
limits:
memory: 2Gi
cpu: 2000m
template:
queryFrontend:
jaegerQuery:
enabled: true
ingress:
type: route
- 1
- This section specifies the deployed object storage back end, which requires a created secret with credentials for access to the object storage.
3.2.1.2. Storage configuration
You can configure object storage for the distributed tracing platform (Tempo) in the TempoStack
custom resource under spec.storage
. You can choose from among several storage providers that are supported.
Parameter | Description | Values | Default value |
---|---|---|---|
| Type of storage to use for the deployment. |
|
|
| Name of the secret that contains the credentials for the set object storage type. | N/A | |
|
CA is the name of a |
Storage provider |
---|
Secret parameters |
|
MinIO |
See MinIO Operator.
|
Amazon S3 |
|
Microsoft Azure Blob Storage |
|
Google Cloud Storage on Google Cloud Platform (GCP) |
|
3.2.1.3. Query configuration options
Two components of the distributed tracing platform (Tempo), the querier and query frontend, manage queries. You can configure both of these components.
The querier component finds the requested trace ID in the ingesters or back-end storage. Depending on the set parameters, the querier component can query both the ingesters and pull bloom or indexes from the back end to search blocks in object storage. The querier component exposes an HTTP endpoint at GET /querier/api/traces/<trace_id>
, but it is not expected to be used directly. Queries must be sent to the query frontend.
Parameter | Description | Values |
---|---|---|
| The simple form of the node-selection constraint. | type: object |
| The number of replicas to be created for the component. | type: integer; format: int32 |
| Component-specific pod tolerations. | type: array |
The query frontend component is responsible for sharding the search space for an incoming query. The query frontend exposes traces via a simple HTTP endpoint: GET /api/traces/<trace_id>
. Internally, the query frontend component splits the blockID
space into a configurable number of shards and then queues these requests. The querier component connects to the query frontend component via a streaming gRPC connection to process these sharded queries.
Parameter | Description | Values |
---|---|---|
| Configuration of the query frontend component. | type: object |
| The simple form of the node selection constraint. | type: object |
| The number of replicas to be created for the query frontend component. | type: integer; format: int32 |
| Pod tolerations specific to the query frontend component. | type: array |
| The options specific to the Jaeger Query component. | type: object |
|
When | type: boolean |
| The options for the Jaeger Query ingress. | type: object |
| The annotations of the ingress object. | type: object |
| The hostname of the ingress object. | type: string |
| The name of an IngressClass cluster resource. Defines which ingress controller serves this ingress resource. | type: string |
| The options for the OpenShift route. | type: object |
|
The termination type. The default is | type: string (enum: insecure, edge, passthrough, reencrypt) |
|
The type of ingress for the Jaeger Query UI. The supported types are | type: string (enum: ingress, route) |
| The monitor tab configuration. | type: object |
|
Enables the monitor tab in the Jaeger console. The | type: boolean |
|
The endpoint to the Prometheus instance that contains the span rate, error, and duration (RED) metrics. For example, | type: string |
Example configuration of the query frontend component in a TempoStack
CR
apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: simplest spec: storage: secret: name: minio type: s3 storageSize: 200M resources: total: limits: memory: 2Gi cpu: 2000m template: queryFrontend: jaegerQuery: enabled: true ingress: route: termination: edge type: route
3.2.1.3.1. Additional resources
3.2.1.4. Configuration of the monitor tab in Jaeger UI
Trace data contains rich information, and the data is normalized across instrumented languages and frameworks. Therefore, request rate, error, and duration (RED) metrics can be extracted from traces. The metrics can be visualized in Jaeger console in the Monitor tab.
The metrics are derived from spans in the OpenTelemetry Collector that are scraped from the Collector by the Prometheus deployed in the user-workload monitoring stack. The Jaeger UI queries these metrics from the Prometheus endpoint and visualizes them.
3.2.1.4.1. OpenTelemetry Collector configuration
The OpenTelemetry Collector requires configuration of the spanmetrics
connector that derives metrics from traces and exports the metrics in the Prometheus format.
OpenTelemetry Collector custom resource for span RED
kind: OpenTelemetryCollector apiVersion: opentelemetry.io/v1alpha1 metadata: name: otel spec: mode: deployment observability: metrics: enableMetrics: true 1 config: | connectors: spanmetrics: 2 metrics_flush_interval: 15s receivers: otlp: 3 protocols: grpc: http: exporters: prometheus: 4 endpoint: 0.0.0.0:8889 add_metric_suffixes: false resource_to_telemetry_conversion: enabled: true # by default resource attributes are dropped otlp: endpoint: "tempo-simplest-distributor:4317" tls: insecure: true service: pipelines: traces: receivers: [otlp] exporters: [otlp, spanmetrics] 5 metrics: receivers: [spanmetrics] 6 exporters: [prometheus]
- 1
- Creates the
ServiceMonitor
custom resource to enable scraping of the Prometheus exporter. - 2
- The Spanmetrics connector receives traces and exports metrics.
- 3
- The OTLP receiver to receive spans in the OpenTelemetry protocol.
- 4
- The Prometheus exporter is used to export metrics in the Prometheus format.
- 5
- The Spanmetrics connector is configured as exporter in traces pipeline.
- 6
- The Spanmetrics connector is configured as receiver in metrics pipeline.
3.2.1.4.2. Tempo configuration
The TempoStack
custom resource must specify the following: the Monitor tab is enabled, and the Prometheus endpoint is set to the Thanos querier service to query the data from the user-defined monitoring stack.
TempoStack custom resource with the enabled Monitor tab
kind: TempoStack apiVersion: tempo.grafana.com/v1alpha1 metadata: name: simplest spec: template: queryFrontend: jaegerQuery: enabled: true monitorTab: enabled: true 1 prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 2 ingress: type: route
3.2.1.4.3. Span RED metrics and alerting rules
The metrics generated by the spanmetrics
connector are usable with alerting rules. For example, for alerts about a slow service or to define service level objectives (SLOs), the connector creates a duration_bucket
histogram and the calls
counter metric. These metrics have labels that identify the service, API name, operation type, and other attributes.
Label | Description | Values |
---|---|---|
|
Service name set by the |
|
| Name of the operation. |
|
| Identifies the server, client, messaging, or internal operation. |
|
Example PrometheusRule
CR that defines an alerting rule for SLO when not serving 95% of requests within 2000ms on the front-end service
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: span-red
spec:
groups:
- name: server-side-latency
rules:
- alert: SpanREDFrontendAPIRequestLatency
expr: histogram_quantile(0.95, sum(rate(duration_bucket{service_name="frontend", span_kind="SPAN_KIND_SERVER"}[5m])) by (le, service_name, span_name)) > 2000 1
labels:
severity: Warning
annotations:
summary: "High request latency on {{$labels.service_name}} and {{$labels.span_name}}"
description: "{{$labels.instance}} has 95th request latency above 2s (current value: {{$value}}s)"
- 1
- The expression for checking if 95% of the front-end server response time values are below 2000 ms. The time range (
[5m]
) must be at least four times the scrape interval and long enough to accommodate a change in the metric.
3.2.1.5. Multitenancy
Multitenancy with authentication and authorization is provided in the Tempo Gateway service. The authentication uses OpenShift OAuth and the Kubernetes TokenReview
API. The authorization uses the Kubernetes SubjectAccessReview
API.
The Tempo Gateway service supports ingestion of traces only via the OTLP/gRPC. The OTLP/HTTP is not supported.
Sample Tempo CR with two tenants, dev
and prod
apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: simplest spec: tenants: mode: openshift 1 authentication: 2 - tenantName: dev 3 tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa" 4 - tenantName: prod tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb" template: gateway: enabled: true 5 queryFrontend: jaegerQuery: enabled: true
- 1
- Must be set to
openshift
. - 2
- The list of tenants.
- 3
- The tenant name. Must be provided in the
X-Scope-OrgId
header when ingesting the data. - 4
- A unique tenant ID.
- 5
- Enables a gateway that performs authentication and authorization. The Jaeger UI is exposed at
http://<gateway-ingress>/api/traces/v1/<tenant-name>/search
.
The authorization configuration uses the ClusterRole
and ClusterRoleBinding
of the Kubernetes Role-Based Access Control (RBAC). By default, no users have read or write permissions.
Sample of the read RBAC configuration that allows authenticated users to read the trace data of the dev
and prod
tenants
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: tempostack-traces-reader rules: - apiGroups: - 'tempo.grafana.com' resources: 1 - dev - prod resourceNames: - traces verbs: - 'get' 2 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tempostack-traces-reader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: tempostack-traces-reader subjects: - kind: Group apiGroup: rbac.authorization.k8s.io name: system:authenticated 3
Sample of the write RBAC configuration that allows the otel-collector
service account to write the trace data for the dev
tenant
apiVersion: v1 kind: ServiceAccount metadata: name: otel-collector 1 namespace: otel --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: tempostack-traces-write rules: - apiGroups: - 'tempo.grafana.com' resources: 2 - dev resourceNames: - traces verbs: - 'create' 3 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tempostack-traces roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: tempostack-traces-write subjects: - kind: ServiceAccount name: otel-collector namespace: otel
Trace data can be sent to the Tempo instance from the OpenTelemetry Collector that uses the service account with RBAC for writing the data.
Sample OpenTelemetry CR configuration
apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: cluster-collector namespace: tracing-system spec: mode: deployment serviceAccount: otel-collector config: | extensions: bearertokenauth: filename: "/var/run/secrets/kubernetes.io/serviceaccount/token" exporters: otlp/dev: endpoint: tempo-simplest-gateway.tempo.svc.cluster.local:8090 tls: insecure: false ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt" auth: authenticator: bearertokenauth headers: X-Scope-OrgID: "dev" service: extensions: [bearertokenauth] pipelines: traces: exporters: [otlp/dev]
3.2.2. Configuring monitoring and alerts
The Tempo Operator supports monitoring and alerts about each TempoStack component such as distributor, ingester, and so on, and exposes upgrade and operational metrics about the Operator itself.
3.2.2.1. Configuring the TempoStack metrics and alerts
You can enable metrics and alerts of TempoStack instances.
Prerequisites
- Monitoring for user-defined projects is enabled in the cluster.
Procedure
To enable metrics of a TempoStack instance, set the
spec.observability.metrics.createServiceMonitors
field totrue
:apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: <name> spec: observability: metrics: createServiceMonitors: true
To enable alerts for a TempoStack instance, set the
spec.observability.metrics.createPrometheusRules
field totrue
:apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: <name> spec: observability: metrics: createPrometheusRules: true
Verification
You can use the Administrator view of the web console to verify successful configuration:
-
Go to Observe
Targets, filter for Source: User, and check that ServiceMonitors in the format tempo-<instance_name>-<component>
have the Up status. -
To verify that alerts are set up correctly, go to Observe
Alerting Alerting rules, filter for Source: User, and check that the Alert rules for the TempoStack instance components are available.
3.2.2.1.1. Additional resources
3.2.2.2. Configuring the Tempo Operator metrics and alerts
When installing the Tempo Operator from the web console, you can select the Enable Operator recommended cluster monitoring on this Namespace checkbox, which enables creating metrics and alerts of the Tempo Operator.
If the checkbox was not selected during installation, you can manually enable metrics and alerts even after installing the Tempo Operator.
Procedure
-
Add the
openshift.io/cluster-monitoring: "true"
label in the project where the Tempo Operator is installed, which isopenshift-tempo-operator
by default.
Verification
You can use the Administrator view of the web console to verify successful configuration:
-
Go to Observe
Targets, filter for Source: Platform, and search for tempo-operator
, which must have the Up status. -
To verify that alerts are set up correctly, go to Observe
Alerting Alerting rules, filter for Source: Platform, and locate the Alert rules for the Tempo Operator.
3.3. Upgrading
For version upgrades, the Tempo Operator uses the Operator Lifecycle Manager (OLM), which controls installation, upgrade, and role-based access control (RBAC) of Operators in a cluster.
The OLM runs in the OpenShift Container Platform by default. The OLM queries for available Operators as well as upgrades for installed Operators.
When the Tempo Operator is upgraded to the new version, it scans for running TempoStack instances that it manages and upgrades them to the version corresponding to the Operator’s new version.
3.3.1. Additional resources
3.4. Removing
The steps for removing the Red Hat OpenShift distributed tracing platform (Tempo) from an OpenShift Container Platform cluster are as follows:
- Shut down all distributed tracing platform (Tempo) pods.
- Remove any TempoStack instances.
- Remove the Tempo Operator.
3.4.1. Removing by using the web console
You can remove a TempoStack instance in the Administrator view of the web console.
Prerequisites
-
You are logged in to the OpenShift Container Platform web console as a cluster administrator with the
cluster-admin
role. -
For Red Hat OpenShift Dedicated, you must be logged in using an account with the
dedicated-admin
role.
Procedure
-
Go to Operators
Installed Operators Tempo Operator TempoStack. -
To remove the TempoStack instance, select
Delete TempoStack Delete. - Optional: Remove the Tempo Operator.
3.4.2. Removing by using the CLI
You can remove a TempoStack instance on the command line.
Prerequisites
An active OpenShift CLI (
oc
) session by a cluster administrator with thecluster-admin
role.Tip-
Ensure that your OpenShift CLI (
oc
) version is up to date and matches your OpenShift Container Platform version. Run
oc login
:$ oc login --username=<your_username>
-
Ensure that your OpenShift CLI (
Procedure
Get the name of the TempoStack instance by running the following command:
$ oc get deployments -n <project_of_tempostack_instance>
Remove the TempoStack instance by running the following command:
$ oc delete tempo <tempostack_instance_name> -n <project_of_tempostack_instance>
- Optional: Remove the Tempo Operator.
Verification
Run the following command to verify that the TempoStack instance is not found in the output, which indicates its successful removal:
$ oc get deployments -n <project_of_tempostack_instance>