Chapter 1. Connectivity Link observability

Procedure

1. Enable tracing in OpenShift Service Mesh by configuring your Telemetry custom resource (CR) as follows:

   Example OpenShift Service Mesh Telemetry CR with tracing

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: gateway-system
   spec:
     tracing:
     - providers:
       - name: tempo-otlp
       randomSamplingPercentage: 100
   # ...

   Copy to Clipboard Toggle word wrap

2. Apply the configuration by running the following command:

   $ kubectl apply -f mesh-default.yaml

   Copy to Clipboard Toggle word wrap

3. Configure a tracing extension provider for OpenShift Service Mesh in your Istio CR by adding a list value to the spec.values.meshConfig.extensionProviders parameter. Ensure that you also add the otel port and service information:

   Example Istio CR with tracing extension provider

   apiVersion: operator.istio.io/v1alpha1
   kind: Istio
   metadata:
     name: default
   spec:
     namespace: gateway-system
     values:
       meshConfig:
         defaultConfig:
           tracing: {}
         enableTracing: true
         extensionProviders:
         - name: tempo-otlp
           opentelemetry:
             port: 4317
             service: tempo.tempo.svc.cluster.local
   # ...

   Copy to Clipboard Toggle word wrap

   Important

   If you are setting the controller manually, you must set the OpenTelemetry Collector protocol in the Service CR port name and appProtocol fields. For example, when using gRPC, the port name should begin with grpc- or the appProtocol should be grpc:

   kind: Service
   apiVersion: v1
   metadata:
     name: otel-collector
   spec:
     ports:
       - name: otlp-grpc
         port: 4317
         targetPort: 4317
         appProtocol: grpc
     selector:
       app: otel-collector
   # ...

   Copy to Clipboard Toggle word wrap

4. Apply the configuration by running the following command:

   $ kubectl apply -f istio.yaml

   Copy to Clipboard Toggle word wrap

5. Optional. If you want to collect Authorino and Limitador traces in a different location than your Kaudrant traces, complete the following steps:

   1. Enable request tracing in your Authorino custom resource (CR) and send authentication and authorization traces to the central collector as follows:

      Example Authorino CR with request tracing

      apiVersion: operator.authorino.kuadrant.io/v1beta1
      kind: Authorino
      metadata:
        name: authorino
      spec:
        tracing:
          endpoint: rpc://authorino-collector:4317
          insecure: true
      # ...

      Copy to Clipboard Toggle word wrap

      Set insecure to true to skip TLS certificate verification in development environments. Set to false for production environments.

   2. Apply the configuration by running the following command:

      $ kubectl apply -f authorino.yaml

      Copy to Clipboard Toggle word wrap

   3. Enable request tracing in your Limitador CR and send rate limit traces to the central collector as follows:

      Example Limitador CR with request tracing

      apiVersion: limitador.kuadrant.io/v1alpha1
      kind: Limitador
      metadata:
        name: limitador
      spec:
        tracing:
          endpoint: rpc://limitador-collector:4317
          insecure: true
      # ...

      Copy to Clipboard Toggle word wrap

      Set insecure to true to skip TLS certificate verification in development environments. Set to false for production environments.

   4. Apply the configuration by running the following command:

      $ kubectl apply -f limitador.yaml

      Copy to Clipboard Toggle word wrap
      Important

      Trace IDs do not propagate to WebAssembly modules in OpenShift Service Mesh. This means that requests passed to Limitador do not have the relevant parent trace ID. However, if the trace initiation point is outside OpenShift Service Mesh, the parent trace ID is available to Limitador and included in traces. This impacts correlating traces from Limitador with traces from Authorino, the gateway, and any other components in the request path.

6. Configure data-plane tracing in the Kuadrant CR by providing the collector endpoint as shown in the following example:

   Example kuadrant CR

   apiVersion: kuadrant.io/v1beta1
   kind: Kuadrant
   metadata:
     name: kuadrant
     namespace: kuadrant-system
   spec:
     observability:
       dataPlane:
         defaultLevels:
           - debug: "true" # Enable DEBUG level trace filtering
         httpHeaderIdentifier: x-request-id
       tracing:
         defaultEndpoint: rpc://tempo.tempo.svc.cluster.local:4317
         insecure: true

   Copy to Clipboard Toggle word wrap

   where:

   • spec.observability.tracing.defaultEndpoint: The URL of the tracing collector backend. That is, the OpenTelemetry endpoint. The following are supported protocols:

     • rpc:// for gRPC OTLP, port 4317
     • http:// for HTTP OTLP, port 4318

   • `spec.observability.tracing.insecure: Set to true to skip TLS certificate verification in development environments. Set to false for production environments.

     Important

     Point to the collector service, such as the Distributed Tracing Platform, not the query service. The collector receives traces from your applications. The query service is only for viewing traces in the GUI.

7. Apply the configuration by running the following command:

   $ kubectl apply -f kuadrant.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure your Telemetry custom resource (CR) to only log errors by using the following example:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: access-logs-errors-only
     namespace: istio-system
   spec:
     accessLogging:
       - providers:
         - name: envoy
         filter:
           expression: "response.code >= 400"

   Copy to Clipboard Toggle word wrap

2. Configure your Telemetry custom resource (CR) to only log specific routes by using the following example:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: access-logs-api-only
     namespace: istio-system
   spec:
     accessLogging:
       - providers:
         - name: envoy
         filter:
           expression: 'request.url_path.startsWith("/api/")'

   Copy to Clipboard Toggle word wrap

3. Configure your Telemetry custom resource (CR) to exclude health checks by using the following example:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: access-logs-no-healthz
     namespace: istio-system
   spec:
     accessLogging:
       - providers:
         - name: envoy
         filter:
           expression: '!request.url_path.startsWith("/healthz")'

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable request correlation across Connectivity Link components, configure the httpHeaderIdentifier in the Kuadrant CR:

   apiVersion: kuadrant.io/v1beta1
   kind: Kuadrant
   metadata:
     name: kuadrant
     namespace: kuadrant-system
   spec:
     observability:
       dataPlane:
         httpHeaderIdentifier: x-request-id
       tracing:
         defaultEndpoint: rpc://tempo.tempo.svc.cluster.local:4317
         insecure: true

   Copy to Clipboard Toggle word wrap

2. You can correlate logs across all components using the x-request-id, by using the following examples:

   1. View the following Envoy access log entry:

      {
        "start_time": "2026-01-23T15:45:12.345Z",
        "method": "GET",
        "path": "/api/users",
        "response_code": 200,
        "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "route_name": "toystore-route"
      }

      Copy to Clipboard Toggle word wrap

   2. Correlate the following Authorino log entry with the Envoy access log:

      {"level":"info","ts":"2026-01-23T15:45:12.350Z","request_id":"a1b2c3d4-e5f6-7890-abcd-ef1234567890","msg":"auth check succeeded","identity":"alice"}

      Copy to Clipboard Toggle word wrap

   3. Correlate the following Limitador log entry with the Envoy and Authorino logs:

      Request received: ... "x-request-id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" ...

      Copy to Clipboard Toggle word wrap

3. When you combine the three logs, the story of this request_id is:

   • At 15:45:12, a user named alice requested the users' API, /api/users. You can also see the request_id of a1b2c3d4-e5f6-7890-abcd-ef1234567890.
   • The request hit the toystore-route in Envoy.
   • Envoy paused the request and checked authentication with Authorino, see level and info.
   • Authorino verified Alice’s identity: auth check succeeded, identity, alice.
   • Simultaneously, Limitador noted the request to ensure that Alice did not exceed her allowed limit.
   • Finally, Envoy allowed the traffic through, resulting in a 200 response code.