Chapter 1. Release notes for the Red Hat OpenShift distributed tracing platform

1.1. Distributed tracing overview

As a service owner, you can use distributed tracing to instrument your services to gather insights into your service architecture. You can use the Red Hat OpenShift distributed tracing platform for monitoring, network profiling, and troubleshooting the interaction between components in modern, cloud-native, microservices-based applications.

With the distributed tracing platform, you can perform the following functions:

Monitor distributed transactions
Optimize performance and latency
Perform root cause analysis

You can use the Red Hat OpenShift distributed tracing platform (Tempo) in combination with the Red Hat build of OpenTelemetry.

Note

Only supported features are documented. Undocumented features are currently unsupported. If you need assistance with a feature, contact Red Hat’s support.

1.2. Release notes for Red Hat OpenShift distributed tracing platform 3.3.1

The Red Hat OpenShift distributed tracing platform 3.3.1 is a maintenance release with no changes because the Red Hat OpenShift distributed tracing platform is bundled with the Red Hat build of OpenTelemetry that is released with a bug fix.

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.2.1. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

The Red Hat OpenShift distributed tracing platform (Tempo) 3.3.1 is based on the open source Grafana Tempo 2.5.0.

1.2.1.1. Known issues

There is currently a known issue:

Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.2.2. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

The Red Hat OpenShift distributed tracing platform (Jaeger) 3.3.1 is based on the open source Jaeger release 1.57.0.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.2.2.1. Support for the OpenShift Elasticsearch Operator

The Red Hat OpenShift distributed tracing platform (Jaeger) 3.3.1 is supported for use with the OpenShift Elasticsearch Operator 5.6, 5.7, and 5.8.

1.2.2.2. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.3.1, Jaeger and support for Elasticsearch remain deprecated, and both are planned to be removed in a future release. Red Hat will provide support for these components and fixes for CVEs and bugs with critical and higher severity during the current release lifecycle, but these components will no longer receive feature enhancements. The Tempo Operator and the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. Users must adopt the OpenTelemetry and Tempo distributed tracing stack because it is the stack to be enhanced going forward.

In the Red Hat OpenShift distributed tracing platform 3.3.1, the Jaeger agent is deprecated and planned to be removed in the following release. Red Hat will provide bug fixes and support for the Jaeger agent during the current release lifecycle, but the Jaeger agent will no longer receive enhancements and will be removed. The OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry is the preferred Operator for injecting the trace collector agent.

1.2.2.3. Known issues

There are currently known issues:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.3. Release notes for Red Hat OpenShift distributed tracing platform 3.3

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.3.1. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

The Red Hat OpenShift distributed tracing platform (Tempo) 3.3 is based on the open source Grafana Tempo 2.5.0.

1.3.1.1. New features and enhancements

This update introduces the following enhancements:

Support for securing the Jaeger UI and Jaeger APIs with the OpenShift OAuth Proxy. (TRACING-4108)
Support for using the service serving certificates, which are generated by OpenShift Container Platform, on ingestion APIs when multitenancy is disabled. (TRACING-3954)
Support for ingesting by using the OTLP/HTTP protocol when multitenancy is enabled. (TRACING-4171)
Support for the AWS S3 Secure Token authentication. (TRACING-4176)
Support for automatically reloading certificates. (TRACING-4185)
Support for configuring the duration for which service names are available for querying. (TRACING-4214)

1.3.1.2. Bug fixes

This update introduces the following bug fixes:

Before this update, storage certificate names did not support dots. With this update, storage certificate name can contain dots. (TRACING-4348)
Before this update, some users had to select a certificate when accessing the gateway route. With this update, there is no prompt to select a certificate. (TRACING-4431)
Before this update, the gateway component was not scalable. With this update, the gateway component is scalable. (TRACING-4497)
Before this update the Jaeger UI might fail with the 504 Gateway Time-out error when accessed via a route. With this update, users can specify route annotations for increasing timeout, such as haproxy.router.openshift.io/timeout: 3m, when querying large data sets. (TRACING-4511)

1.3.1.3. Known issues

There is currently a known issue:

Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.3.2. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

The Red Hat OpenShift distributed tracing platform (Jaeger) 3.3 is based on the open source Jaeger release 1.57.0.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.3.2.1. Support for the OpenShift Elasticsearch Operator

The Red Hat OpenShift distributed tracing platform (Jaeger) 3.3 is supported for use with the OpenShift Elasticsearch Operator 5.6, 5.7, and 5.8.

1.3.2.2. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.3, Jaeger and support for Elasticsearch remain deprecated, and both are planned to be removed in a future release. Red Hat will provide support for these components and fixes for CVEs and bugs with critical and higher severity during the current release lifecycle, but these components will no longer receive feature enhancements. The Tempo Operator and the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. Users must adopt the OpenTelemetry and Tempo distributed tracing stack because it is the stack to be enhanced going forward.

In the Red Hat OpenShift distributed tracing platform 3.3, the Jaeger agent is deprecated and planned to be removed in the following release. Red Hat will provide bug fixes and support for the Jaeger agent during the current release lifecycle, but the Jaeger agent will no longer receive enhancements and will be removed. The OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry is the preferred Operator for injecting the trace collector agent.

1.3.2.3. Known issues

There are currently known issues:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.4. Release notes for Red Hat OpenShift distributed tracing platform 3.2.2

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.4.1. CVEs

This release fixes the following CVEs:

1.4.2. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

1.4.2.1. Bug fixes

This update introduces the following bug fix:

Before this update, secrets were perpetually generated on OpenShift Container Platform 4.16 because the operator tried to reconcile a new openshift.io/internal-registry-pull-secret-ref annotation for service accounts, causing a loop. With this update, the operator ignores this new annotation. (TRACING-4434)

1.4.2.2. Known issues

There is currently a known issue:

Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.4.3. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.4.3.1. Known issues

There is currently a known issue:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.5. Release notes for Red Hat OpenShift distributed tracing platform 3.2.1

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.5.1. CVEs

This release fixes CVE-2024-25062.

1.5.2. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

1.5.2.1. Known issues

There is currently a known issue:

Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.5.3. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.5.3.1. Known issues

There is currently a known issue:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.6. Release notes for Red Hat OpenShift distributed tracing platform 3.2

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.6.1. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

1.6.1.1. Technology Preview features

This update introduces the following Technology Preview feature:

Support for the Tempo monolithic deployment.

Important

The Tempo monolithic deployment is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

1.6.1.2. New features and enhancements

This update introduces the following enhancements:

Red Hat OpenShift distributed tracing platform (Tempo) 3.2 is based on the open source Grafana Tempo 2.4.1.
Allowing the overriding of resources per component.

1.6.1.3. Bug fixes

This update introduces the following bug fixes:

Before this update, the Jaeger UI only displayed services that sent traces in the previous 15 minutes. With this update, the availability of the service and operation names can be configured by using the following field: spec.template.queryFrontend.jaegerQuery.servicesQueryDuration. (TRACING-3139)
Before this update, the query-frontend pod might get stopped when out-of-memory (OOM) as a result of searching a large trace. With this update, resource limits can be set to prevent this issue. (TRACING-4009)

1.6.1.4. Known issues

There is currently a known issue:

Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.6.2. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.6.2.1. Support for OpenShift Elasticsearch Operator

Red Hat OpenShift distributed tracing platform (Jaeger) 3.2 is supported for use with the OpenShift Elasticsearch Operator 5.6, 5.7, and 5.8.

1.6.2.2. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.2, Jaeger and support for Elasticsearch remain deprecated, and both are planned to be removed in a future release. Red Hat will provide support for these components and fixes for CVEs and bugs with critical and higher severity during the current release lifecycle, but these components will no longer receive feature enhancements. The Tempo Operator and the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. Users must adopt the OpenTelemetry and Tempo distributed tracing stack because it is the stack to be enhanced going forward.

In the Red Hat OpenShift distributed tracing platform 3.2, the Jaeger agent is deprecated and planned to be removed in the following release. Red Hat will provide bug fixes and support for the Jaeger agent during the current release lifecycle, but the Jaeger agent will no longer receive enhancements and will be removed. The OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry is the preferred Operator for injecting the trace collector agent.

1.6.2.3. New features and enhancements

This update introduces the following enhancements for the distributed tracing platform (Jaeger):

Red Hat OpenShift distributed tracing platform (Jaeger) 3.2 is based on the open source Jaeger release 1.57.0.

1.6.2.4. Known issues

There is currently a known issue:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.7. Release notes for Red Hat OpenShift distributed tracing platform 3.1.1

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.7.1. CVEs

This release fixes CVE-2023-39326.

1.7.2. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

1.7.2.1. Known issues

There are currently known issues:

Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.7.3. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.7.3.1. Support for OpenShift Elasticsearch Operator

Red Hat OpenShift distributed tracing platform (Jaeger) 3.1.1 is supported for use with the OpenShift Elasticsearch Operator 5.6, 5.7, and 5.8.

1.7.3.2. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.1.1, Jaeger and support for Elasticsearch remain deprecated, and both are planned to be removed in a future release. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements.

In the Red Hat OpenShift distributed tracing platform 3.1.1, Tempo provided by the Tempo Operator and the OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. The OpenTelemetry and Tempo distributed tracing stack is to be adopted by all users because this will be the stack that will be enhanced going forward.

1.7.3.3. Known issues

There are currently known issues:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.8. Release notes for Red Hat OpenShift distributed tracing platform 3.1

This release of the Red Hat OpenShift distributed tracing platform includes the Red Hat OpenShift distributed tracing platform (Tempo) and the deprecated Red Hat OpenShift distributed tracing platform (Jaeger).

1.8.1. Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is provided through the Tempo Operator.

1.8.1.1. New features and enhancements

This update introduces the following enhancements for the distributed tracing platform (Tempo):

Red Hat OpenShift distributed tracing platform (Tempo) 3.1 is based on the open source Grafana Tempo 2.3.1.
Support for cluster-wide proxy environments.
Support for TraceQL to Gateway component.

1.8.1.2. Bug fixes

This update introduces the following bug fixes for the distributed tracing platform (Tempo):

Before this update, when a TempoStack instance was created with the monitorTab enabled in OpenShift Container Platform 4.15, the required tempo-redmetrics-cluster-monitoring-view ClusterRoleBinding was not created. This update resolves the issue by fixing the Operator RBAC for the monitor tab when the Operator is deployed in an arbitrary namespace. (TRACING-3786)
Before this update, when a TempoStack instance was created on an OpenShift Container Platform cluster with only an IPv6 networking stack, the compactor and ingestor pods ran in the CrashLoopBackOff state, resulting in multiple errors. This update provides support for IPv6 clusters.(TRACING-3226)

1.8.1.3. Known issues

There are currently known issues:

Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.8.2. Red Hat OpenShift distributed tracing platform (Jaeger)

The Red Hat OpenShift distributed tracing platform (Jaeger) is provided through the Red Hat OpenShift distributed tracing platform Operator.

Important

Jaeger does not use FIPS validated cryptographic modules.

1.8.2.1. Support for OpenShift Elasticsearch Operator

Red Hat OpenShift distributed tracing platform (Jaeger) 3.1 is supported for use with the OpenShift Elasticsearch Operator 5.6, 5.7, and 5.8.

1.8.2.2. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.1, Jaeger and support for Elasticsearch remain deprecated, and both are planned to be removed in a future release. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements.

In the Red Hat OpenShift distributed tracing platform 3.1, Tempo provided by the Tempo Operator and the OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. The OpenTelemetry and Tempo distributed tracing stack is to be adopted by all users because this will be the stack that will be enhanced going forward.

1.8.2.3. New features and enhancements

This update introduces the following enhancements for the distributed tracing platform (Jaeger):

Red Hat OpenShift distributed tracing platform (Jaeger) 3.1 is based on the open source Jaeger release 1.53.0.

1.8.2.4. Bug fixes

This update introduces the following bug fix for the distributed tracing platform (Jaeger):

Before this update, the connection target URL for the jaeger-agent container in the jager-query pod was overwritten with another namespace URL in OpenShift Container Platform 4.13. This was caused by a bug in the sidecar injection code in the jaeger-operator, causing nondeterministic jaeger-agent injection. With this update, the Operator prioritizes the Jaeger instance from the same namespace as the target deployment. (TRACING-3722)

1.8.2.5. Known issues

There are currently known issues:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.9. Release notes for Red Hat OpenShift distributed tracing platform 3.0

1.9.1. Component versions in the Red Hat OpenShift distributed tracing platform 3.0

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.51.0
Red Hat OpenShift distributed tracing platform (Tempo)	Tempo	2.3.0

1.9.2. Red Hat OpenShift distributed tracing platform (Jaeger)

1.9.2.1. Deprecated functionality

In the Red Hat OpenShift distributed tracing platform 3.0, Jaeger and support for Elasticsearch are deprecated, and both are planned to be removed in a future release. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements.

In the Red Hat OpenShift distributed tracing platform 3.0, Tempo provided by the Tempo Operator and the OpenTelemetry Collector provided by the Red Hat build of OpenTelemetry are the preferred Operators for distributed tracing collection and storage. The OpenTelemetry and Tempo distributed tracing stack is to be adopted by all users because this will be the stack that will be enhanced going forward.

1.9.2.2. New features and enhancements

This update introduces the following enhancements for the distributed tracing platform (Jaeger):

Support for the ARM architecture.
Support for cluster-wide proxy environments.

1.9.2.3. Bug fixes

This update introduces the following bug fix for the distributed tracing platform (Jaeger):

Before this update, the Red Hat OpenShift distributed tracing platform (Jaeger) Operator used other images than relatedImages. This caused the ImagePullBackOff error in disconnected network environments when launching the jaeger pod because the oc adm catalog mirror command mirrors images specified in relatedImages. This update provides support for disconnected environments when using the oc adm catalog mirror CLI command. (TRACING-3546)

1.9.2.4. Known issues

There is currently a known issue:

Currently, Apache Spark is not supported.
Currently, the streaming deployment via AMQ/Kafka is not supported on the IBM Z and IBM Power architectures.

1.9.3. Red Hat OpenShift distributed tracing platform (Tempo)

1.9.3.1. New features and enhancements

This update introduces the following enhancements for the distributed tracing platform (Tempo):

Support for the ARM architecture.
Support for span request count, duration, and error count (RED) metrics. The metrics can be visualized in the Jaeger console deployed as part of Tempo or in the web console in the Observe menu.

1.9.3.2. Bug fixes

This update introduces the following bug fixes for the distributed tracing platform (Tempo):

Before this update, the TempoStack CRD was not accepting custom CA certificate despite the option to choose CA certificates. This update fixes support for the custom TLS CA option for connecting to object storage. (TRACING-3462)
Before this update, when mirroring the Red Hat OpenShift distributed tracing platform Operator images to a mirror registry for use in a disconnected cluster, the related Operator images for tempo, tempo-gateway, opa-openshift, and tempo-query were not mirrored. This update fixes support for disconnected environments when using the oc adm catalog mirror CLI command. (TRACING-3523)
Before this update, the query frontend service of the Red Hat OpenShift distributed tracing platform was using internal mTLS when gateway was not deployed. This caused endpoint failure errors. This update fixes mTLS when Gateway is not deployed. (TRACING-3510)

1.9.3.3. Known issues

There are currently known issues:

Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

1.10. Release notes for Red Hat OpenShift distributed tracing platform 2.9.2

1.10.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.9.2

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.47.0
Red Hat OpenShift distributed tracing platform (Tempo)	Tempo	2.1.1

1.10.2. CVEs

This release fixes CVE-2023-46234.

1.10.3. Red Hat OpenShift distributed tracing platform (Jaeger)

1.10.3.1. Known issues

There are currently known issues:

Apache Spark is not supported.
The streaming deployment via AMQ/Kafka is unsupported on the IBM Z and IBM Power architectures.

1.10.4. Red Hat OpenShift distributed tracing platform (Tempo)

Important

The Red Hat OpenShift distributed tracing platform (Tempo) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

1.10.4.1. Known issues

There are currently known issues:

Currently, the custom TLS CA option is not implemented for connecting to object storage. (TRACING-3462)
Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)
Currently, the Tempo query frontend service must not use internal mTLS when Gateway is not deployed. This issue does not affect the Jaeger Query API. The workaround is to disable mTLS. (TRACING-3510)
Workaround
Disable mTLS as follows:
1. Open the Tempo Operator ConfigMap for editing by running the following command:
```
$ oc edit configmap tempo-operator-manager-config -n openshift-tempo-operator 1
```
  1
  The project where the Tempo Operator is installed.
2. Disable the mTLS in the Operator configuration by updating the YAML file:
```
data:
  controller_manager_config.yaml: |
    featureGates:
      httpEncryption: false
      grpcEncryption: false
      builtInCertManagement:
        enabled: false
```
3. Restart the Tempo Operator pod by running the following command:
```
$ oc rollout restart deployment.apps/tempo-operator-controller -n openshift-tempo-operator
```

Missing images for running the Tempo Operator in restricted environments. The Red Hat OpenShift distributed tracing platform (Tempo) CSV is missing references to the operand images. (TRACING-3523)

Workaround

Add the Tempo Operator related images in the mirroring tool to mirror the images to the registry:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 20
storageConfig:
  local:
    path: /home/user/images
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
    packages:
    - name: tempo-product
      channels:
      - name: stable
  additionalImages:
  - name: registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a
  - name: registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23
  - name: registry.redhat.io/rhosdt/tempo-gateway-opa-rhel8@sha256:8cd134deca47d6817b26566e272e6c3f75367653d589f5c90855c59b2fab01e9
  - name: registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e

1.11. Release notes for Red Hat OpenShift distributed tracing platform 2.9.1

1.11.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.9.1

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.47.0
Red Hat OpenShift distributed tracing platform (Tempo)	Tempo	2.1.1

1.11.2. CVEs

This release fixes CVE-2023-44487.

1.11.3. Red Hat OpenShift distributed tracing platform (Jaeger)

1.11.3.1. Known issues

There are currently known issues:

Apache Spark is not supported.
The streaming deployment via AMQ/Kafka is unsupported on the IBM Z and IBM Power architectures.

1.11.4. Red Hat OpenShift distributed tracing platform (Tempo)

Important

The Red Hat OpenShift distributed tracing platform (Tempo) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

1.11.4.1. Known issues

There are currently known issues:

Currently, the custom TLS CA option is not implemented for connecting to object storage. (TRACING-3462)
Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)
Currently, the Tempo query frontend service must not use internal mTLS when Gateway is not deployed. This issue does not affect the Jaeger Query API. The workaround is to disable mTLS. (TRACING-3510)
Workaround
Disable mTLS as follows:
1. Open the Tempo Operator ConfigMap for editing by running the following command:
```
$ oc edit configmap tempo-operator-manager-config -n openshift-tempo-operator 1
```
  1
  The project where the Tempo Operator is installed.
2. Disable the mTLS in the Operator configuration by updating the YAML file:
```
data:
  controller_manager_config.yaml: |
    featureGates:
      httpEncryption: false
      grpcEncryption: false
      builtInCertManagement:
        enabled: false
```
3. Restart the Tempo Operator pod by running the following command:
```
$ oc rollout restart deployment.apps/tempo-operator-controller -n openshift-tempo-operator
```

Missing images for running the Tempo Operator in restricted environments. The Red Hat OpenShift distributed tracing platform (Tempo) CSV is missing references to the operand images. (TRACING-3523)

Workaround

Add the Tempo Operator related images in the mirroring tool to mirror the images to the registry:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 20
storageConfig:
  local:
    path: /home/user/images
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
    packages:
    - name: tempo-product
      channels:
      - name: stable
  additionalImages:
  - name: registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a
  - name: registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23
  - name: registry.redhat.io/rhosdt/tempo-gateway-opa-rhel8@sha256:8cd134deca47d6817b26566e272e6c3f75367653d589f5c90855c59b2fab01e9
  - name: registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e

1.12. Release notes for Red Hat OpenShift distributed tracing platform 2.9

1.12.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.9

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.47.0
Red Hat OpenShift distributed tracing platform (Tempo)	Tempo	2.1.1

1.12.2. Red Hat OpenShift distributed tracing platform (Jaeger)

1.12.2.1. Bug fixes

Before this update, connection was refused due to a missing gRPC port on the jaeger-query deployment. This issue resulted in transport: Error while dialing: dial tcp :16685: connect: connection refused error message. With this update, the Jaeger Query gRPC port (16685) is successfully exposed on the Jaeger Query service. (TRACING-3322)
Before this update, the wrong port was exposed for jaeger-production-query, resulting in refused connection. With this update, the issue is fixed by exposing the Jaeger Query gRPC port (16685) on the Jaeger Query deployment. (TRACING-2968)
Before this update, when deploying Service Mesh on single-node OpenShift clusters in disconnected environments, the Jaeger pod frequently went into the Pending state. With this update, the issue is fixed. (TRACING-3312)
Before this update, the Jaeger Operator pod restarted with the default memory value due to the reason: OOMKilled error message. With this update, this issue is fixed by removing the resource limits. (TRACING-3173)

1.12.2.2. Known issues

There are currently known issues:

Apache Spark is not supported.
The streaming deployment via AMQ/Kafka is unsupported on the IBM Z and IBM Power architectures.

1.12.3. Red Hat OpenShift distributed tracing platform (Tempo)

Important

The Red Hat OpenShift distributed tracing platform (Tempo) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

1.12.3.1. New features and enhancements

This release introduces the following enhancements for the distributed tracing platform (Tempo):

Support the operator maturity Level IV, Deep Insights, which enables upgrading, monitoring, and alerting of the TempoStack instances and the Tempo Operator.
Add Ingress and Route configuration for the Gateway.
Support the managed and unmanaged states in the TempoStack custom resource.
Expose the following additional ingestion protocols in the Distributor service: Jaeger Thrift binary, Jaeger Thrift compact, Jaeger gRPC, and Zipkin. When the Gateway is enabled, only the OpenTelemetry protocol (OTLP) gRPC is enabled.
Expose the Jaeger Query gRPC endpoint on the Query Frontend service.
Support multitenancy without Gateway authentication and authorization.

1.12.3.2. Bug fixes

Before this update, the Tempo Operator was not compatible with disconnected environments. With this update, the Tempo Operator supports disconnected environments. (TRACING-3145)
Before this update, the Tempo Operator with TLS failed to start on OpenShift Container Platform. With this update, the mTLS communication is enabled between Tempo components, the Operand starts successfully, and the Jaeger UI is accessible. (TRACING-3091)
Before this update, the resource limits from the Tempo Operator caused error messages such as reason: OOMKilled. With this update, the resource limits for the Tempo Operator are removed to avoid such errors. (TRACING-3204)

1.12.3.3. Known issues

There are currently known issues:

Currently, the custom TLS CA option is not implemented for connecting to object storage. (TRACING-3462)
Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)
Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)
Currently, the Tempo query frontend service must not use internal mTLS when Gateway is not deployed. This issue does not affect the Jaeger Query API. The workaround is to disable mTLS. (TRACING-3510)
Workaround
Disable mTLS as follows:
1. Open the Tempo Operator ConfigMap for editing by running the following command:
```
$ oc edit configmap tempo-operator-manager-config -n openshift-tempo-operator 1
```
  1
  The project where the Tempo Operator is installed.
2. Disable the mTLS in the Operator configuration by updating the YAML file:
```
data:
  controller_manager_config.yaml: |
    featureGates:
      httpEncryption: false
      grpcEncryption: false
      builtInCertManagement:
        enabled: false
```
3. Restart the Tempo Operator pod by running the following command:
```
$ oc rollout restart deployment.apps/tempo-operator-controller -n openshift-tempo-operator
```

Missing images for running the Tempo Operator in restricted environments. The Red Hat OpenShift distributed tracing platform (Tempo) CSV is missing references to the operand images. (TRACING-3523)

Workaround

Add the Tempo Operator related images in the mirroring tool to mirror the images to the registry:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 20
storageConfig:
  local:
    path: /home/user/images
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
    packages:
    - name: tempo-product
      channels:
      - name: stable
  additionalImages:
  - name: registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a
  - name: registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23
  - name: registry.redhat.io/rhosdt/tempo-gateway-opa-rhel8@sha256:8cd134deca47d6817b26566e272e6c3f75367653d589f5c90855c59b2fab01e9
  - name: registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e

1.13. Release notes for Red Hat OpenShift distributed tracing platform 2.8

1.13.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.8

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.42
Red Hat OpenShift distributed tracing platform (Tempo)	Tempo	0.1.0

1.13.2. Technology Preview features

This release introduces support for the Red Hat OpenShift distributed tracing platform (Tempo) as a Technology Preview feature for Red Hat OpenShift distributed tracing platform.

Important

The Red Hat OpenShift distributed tracing platform (Tempo) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The feature uses version 0.1.0 of the Red Hat OpenShift distributed tracing platform (Tempo) and version 2.0.1 of the upstream distributed tracing platform (Tempo) components.

You can use the distributed tracing platform (Tempo) to replace Jaeger so that you can use S3-compatible storage instead of ElasticSearch. Most users who use the distributed tracing platform (Tempo) instead of Jaeger will not notice any difference in functionality because the distributed tracing platform (Tempo) supports the same ingestion and query protocols as Jaeger and uses the same user interface.

If you enable this Technology Preview feature, note the following limitations of the current implementation:

The distributed tracing platform (Tempo) currently does not support disconnected installations. (TRACING-3145)
When you use the Jaeger user interface (UI) with the distributed tracing platform (Tempo), the Jaeger UI lists only services that have sent traces within the last 15 minutes. For services that have not sent traces within the last 15 minutes, those traces are still stored even though they are not visible in the Jaeger UI. (TRACING-3139)

Expanded support for the Tempo Operator is planned for future releases of the Red Hat OpenShift distributed tracing platform. Possible additional features might include support for TLS authentication, multitenancy, and multiple clusters. For more information about the Tempo Operator, see the Tempo community documentation.

1.13.3. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.14. Release notes for Red Hat OpenShift distributed tracing platform 2.7

1.14.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.7

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.39

1.14.2. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.15. Release notes for Red Hat OpenShift distributed tracing platform 2.6

1.15.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.6

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.38

1.15.2. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.16. Release notes for Red Hat OpenShift distributed tracing platform 2.5

1.16.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.5

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.36

1.16.2. New features and enhancements

This release introduces support for ingesting OpenTelemetry protocol (OTLP) to the Red Hat OpenShift distributed tracing platform (Jaeger) Operator. The Operator now automatically enables the OTLP ports:

Port 4317 for the OTLP gRPC protocol.
Port 4318 for the OTLP HTTP protocol.

This release also adds support for collecting Kubernetes resource attributes to the Red Hat build of OpenTelemetry Operator.

1.16.3. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.17. Release notes for Red Hat OpenShift distributed tracing platform 2.4

1.17.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.4

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.34.1

1.17.2. New features and enhancements

This release adds support for auto-provisioning certificates using the OpenShift Elasticsearch Operator.

Self-provisioning by using the Red Hat OpenShift distributed tracing platform (Jaeger) Operator to call the OpenShift Elasticsearch Operator during installation.

+

Important

When upgrading to the Red Hat OpenShift distributed tracing platform 2.4, the Operator recreates the Elasticsearch instance, which might take five to ten minutes. Distributed tracing will be down and unavailable for that period.

1.17.3. Technology Preview features

Creating the Elasticsearch instance and certificates first and then configuring the distributed tracing platform (Jaeger) to use the certificate is a Technology Preview for this release.

1.17.4. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.18. Release notes for Red Hat OpenShift distributed tracing platform 2.3

1.18.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.3.1

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.30.2

1.18.2. Component versions in the Red Hat OpenShift distributed tracing platform 2.3.0

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.30.1

1.18.3. New features and enhancements

With this release, the Red Hat OpenShift distributed tracing platform (Jaeger) Operator is now installed to the openshift-distributed-tracing namespace by default. Before this update, the default installation had been in the openshift-operators namespace.

1.18.4. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.19. Release notes for Red Hat OpenShift distributed tracing platform 2.2

1.19.1. Technology Preview features

The unsupported OpenTelemetry Collector components included in the 2.1 release are removed.

1.19.2. Bug fixes

This release of the Red Hat OpenShift distributed tracing platform addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.20. Release notes for Red Hat OpenShift distributed tracing platform 2.1

1.20.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.1

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.29.1

1.20.2. Technology Preview features

This release introduces a breaking change to how to configure certificates in the OpenTelemetry custom resource file. With this update, the ca_file moves under tls in the custom resource, as shown in the following examples.

CA file configuration for OpenTelemetry version 0.33

spec:
  mode: deployment
  config: |
    exporters:
      jaeger:
        endpoint: jaeger-production-collector-headless.tracing-system.svc:14250
        ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"

CA file configuration for OpenTelemetry version 0.41.1

spec:
  mode: deployment
  config: |
    exporters:
      jaeger:
        endpoint: jaeger-production-collector-headless.tracing-system.svc:14250
        tls:
          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"

1.20.3. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.21. Release notes for Red Hat OpenShift distributed tracing platform 2.0

1.21.1. Component versions in the Red Hat OpenShift distributed tracing platform 2.0

Operator	Component	Version
Red Hat OpenShift distributed tracing platform (Jaeger)	Jaeger	1.28.0

1.21.2. New features and enhancements

This release introduces the following new features and enhancements:

Rebrands Red Hat OpenShift Jaeger as the Red Hat OpenShift distributed tracing platform.
Updates Red Hat OpenShift distributed tracing platform (Jaeger) Operator to Jaeger 1.28. Going forward, the Red Hat OpenShift distributed tracing platform will only support the stable Operator channel. Channels for individual releases are no longer supported.
Adds support for OpenTelemetry protocol (OTLP) to the Query service.
Introduces a new distributed tracing icon that appears in the OperatorHub.
Includes rolling updates to the documentation to support the name change and new features.

1.21.3. Technology Preview features

This release adds the Red Hat build of OpenTelemetry as a Technology Preview, which you install using the Red Hat build of OpenTelemetry Operator. Red Hat build of OpenTelemetry is based on the OpenTelemetry APIs and instrumentation. The Red Hat build of OpenTelemetry includes the OpenTelemetry Operator and Collector. You can use the Collector to receive traces in the OpenTelemetry or Jaeger protocol and send the trace data to the Red Hat OpenShift distributed tracing platform. Other capabilities of the Collector are not supported at this time. The OpenTelemetry Collector allows developers to instrument their code with vendor agnostic APIs, avoiding vendor lock-in and enabling a growing ecosystem of observability tooling.

1.21.4. Bug fixes

This release addresses Common Vulnerabilities and Exposures (CVEs) and bug fixes.

1.22. Getting support

If you experience difficulty with a procedure described in this documentation, or with OpenShift Container Platform in general, visit the Red Hat Customer Portal.

From the Customer Portal, you can:

Search or browse through the Red Hat Knowledgebase of articles and solutions relating to Red Hat products.
Submit a support case to Red Hat Support.
Access other product documentation.

To identify issues with your cluster, you can use Insights in OpenShift Cluster Manager. Insights provides details about issues and, if available, information on how to solve a problem.

If you have a suggestion for improving this documentation or have found an error, submit a Jira issue for the most relevant documentation component. Please provide specific details, such as the section name and OpenShift Container Platform version.

1.23. Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.