Chapter 4. Configuring Red Hat OpenShift distributed tracing data collection with Service Mesh
You can integrate Red Hat OpenShift Service Mesh with Red Hat OpenShift distributed tracing data collection to instrument, generate, collect, and export OpenTelemetry traces, metrics, and logs to analyze and understand your software’s performance and behavior.
Prerequisites
- Tempo Operator is installed. See: Installing the Tempo Operator.
- Red Hat OpenShift distributed tracing data collection Operator is installed. See: Installing the Red Hat build of OpenTelemetry
-
A TempoStack is installed and configured in a
tempo
namespace. See: Installing a TempoStack instance. - An Istio instance is created.
- An Istio CNI instance is created.
Procedure
Navigate to the Red Hat OpenShift distributed tracing data collection Operator and install the
OpenTelemetryCollector
resource in theistio-system
namespace:Example OpenTelemetry Collector in
istio-system
namespacekind: OpenTelemetryCollector apiVersion: opentelemetry.io/v1beta1 metadata: name: otel namespace: istio-system spec: observability: metrics: {} deploymentUpdateStrategy: {} config: exporters: otlp: endpoint: 'tempo-sample-distributor.tempo.svc.cluster.local:4317' tls: insecure: true receivers: otlp: protocols: grpc: endpoint: '0.0.0.0:4317' http: {} service: pipelines: traces: exporters: - otlp receivers: - otlp
Configure Red Hat OpenShift Service Mesh to enable tracing, and define the distributed tracing data collection tracing providers in your
meshConfig
:Example enabling tracing and defining tracing providers
apiVersion: sailoperator.io/v1alpha1 kind: Istio metadata: # ... name: default spec: namespace: istio-system # ... values: meshConfig: enableTracing: true extensionProviders: - name: otel-tracing opentelemetry: port: 4317 service: otel-collector.istio-system.svc.cluster.local 1
- 1
- The
service
field is theOpenTelemetry
collector service in theistio-system
namespace.
Create an Istio Telemetry resource to enable tracers defined in
spec.values.meshConfig.ExtensionProviders
:Example Istio Telemetry resource
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: otel-demo namespace: istio-system spec: tracing: - providers: - name: otel-tracing randomSamplingPercentage: 100
NoteOnce you verify that you can see traces, lower the
randomSamplingPercentage
value or set it todefault
to reduce the number of requests.Deploy the
bookinfo
application inbookinfo
namespace:oc create ns bookinfo
oc label ns <namespace_name> istio.io/rev= 1
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo
- 1
- If you named your Istio resource
default
and are using theInPlace
upgrade strategy, useoc label ns bookinfo istio-injection=enabled
.
NoteTo find your
<revision-name>
, run the following command:oc get istiorevisions.sailoperator.io
Sample output:
NAME TYPE READY STATUS IN USE VERSION AGE default-v1-23-0 Local True Healthy False v1.23.0 3m33s
Generate traffic to the
productpage
pod to generate traces:$ oc exec -it -n bookinfo deployments/productpage-v1 -c istio-proxy -- curl localhost:9080/productpage
Validate the integration by running the following command to see traces in the UI:
$ kubectl get routes -n tempo tempo-sample-query-frontend