Chapter 3. Distributed tracing and Service Mesh
3.1. Configuring Red Hat OpenShift distributed tracing platform with Service Mesh Copy linkLink copied to clipboard!
Integrating Red Hat OpenShift distributed tracing platform with Red Hat OpenShift Service Mesh is made of up two parts: Red Hat OpenShift distributed tracing platform (Tempo) and Red Hat OpenShift distributed tracing data collection.
- Red Hat OpenShift distributed tracing platform (Tempo)
Provides distributed tracing to monitor and troubleshoot transactions in complex distributed systems. Tempo is based on the open source Grafana Tempo project.
For more about information about distributed tracing platform (Tempo), its features, installation, and configuration, see: Red Hat OpenShift distributed tracing platform (Tempo).
- Red Hat OpenShift distributed tracing data collection
Is based on the open source OpenTelemetry project, which aims to provide unified, standardized, and vendor-neutral telemetry data collection for cloud-native software. Red Hat OpenShift distributed tracing data collection product provides support for deploying and managing the OpenTelemetry Collector and simplifying the workload instrumentation.
The OpenTelemetry Collector can receive, process, and forward telemetry data in multiple formats, making it the ideal component for telemetry processing and interoperability between telemetry systems. The Collector provides a unified solution for collecting and processing metrics, traces, and logs.
For more information about distributed tracing data collection, its features, installation, and configuration, see: Red Hat OpenShift distributed tracing data collection.
3.1.1. Configuring Red Hat OpenShift distributed tracing data collection with Service Mesh Copy linkLink copied to clipboard!
You can integrate Red Hat OpenShift Service Mesh with Red Hat OpenShift distributed tracing data collection to instrument, generate, collect, and export OpenTelemetry traces, metrics, and logs to analyze and understand your software’s performance and behavior.
Prerequisites
- Tempo Operator is installed. See: Installing the Tempo Operator.
- Red Hat OpenShift distributed tracing data collection Operator is installed. See: Installing the Red Hat build of OpenTelemetry
-
A TempoStack is installed and configured in a
tempo
namespace. See: Installing a TempoStack instance. - An Istio instance is created.
- An Istio CNI instance is created.
Procedure
Navigate to the Red Hat OpenShift distributed tracing data collection Operator and install the
OpenTelemetryCollector
resource in theistio-system
namespace:Example OpenTelemetry Collector in
istio-system
namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure Red Hat OpenShift Service Mesh to enable tracing, and define the distributed tracing data collection tracing providers in your
meshConfig
:Example enabling tracing and defining tracing providers
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The
service
field is theOpenTelemetry
collector service in theistio-system
namespace.
Create an Istio Telemetry resource to enable tracers defined in
spec.values.meshConfig.ExtensionProviders
:Example Istio Telemetry resource
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteOnce you verify that you can see traces, lower the
randomSamplingPercentage
value or set it todefault
to reduce the number of requests.Create the
bookinfo
namespace by running the following command:oc create ns bookinfo
$ oc create ns bookinfo
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Depending on the update strategy you are using, enable sidecar injection in the namespace by running the appropriate commands:
If you are using the
InPlace
update strategy, run the following command:oc label namespace curl istio-injection=enabled
$ oc label namespace curl istio-injection=enabled
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you are using the
RevisionBased
update strategy, run the following commands:Display the revision name by running the following command:
oc get istiorevisions.sailoperator.io
$ oc get istiorevisions.sailoperator.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME TYPE READY STATUS IN USE VERSION AGE default Local True Healthy True v1.24.3 3m33s
NAME TYPE READY STATUS IN USE VERSION AGE default Local True Healthy True v1.24.3 3m33s
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Label the namespace with the revision name to enable sidecar injection by running the following command:
oc label namespace curl istio.io/rev=default
$ oc label namespace curl istio.io/rev=default
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Deploy the
bookinfo
application in thebookinfo
namespace by running the following command:oc apply -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo
$ oc apply -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Generate traffic to the
productpage
pod to generate traces:oc exec -it -n bookinfo deployments/productpage-v1 -c istio-proxy -- curl localhost:9080/productpage
$ oc exec -it -n bookinfo deployments/productpage-v1 -c istio-proxy -- curl localhost:9080/productpage
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Validate the integration by running the following command to see traces in the UI:
oc get routes -n tempo tempo-sample-query-frontend
$ oc get routes -n tempo tempo-sample-query-frontend
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe OpenShift route for Jaeger UI must be created in the Tempo namespace. You can either manually create it for the
tempo-sample-query-frontend
service, or update theTempo
custom resource with.spec.template.queryFrontend.jaegerQuery.ingress.type: route
.