Connectivity Link observability guide

Procedure

1. Verify that user workload monitoring is configured correctly in your OpenShift cluster as follows:

   kubectl get configmap cluster-monitoring-config -n openshift-monitoring -o jsonpath='{.data.config\.yaml}'|grep enableUserWorkload

   Copy to Clipboard Toggle word wrap

   The expected output is enableUserWorkload: true.

2. Install the Connectivity Link, Gateway, and Grafana component metrics and configuration as follows:

   kubectl apply -k https://github.com/Kuadrant/kuadrant-operator/config/install/configure/observability?ref=v1.1.0

   Copy to Clipboard Toggle word wrap

3. From the root directory of your Kuadrant Operator repository, configure the OpenShift thanos-query instance as a data source in Grafana as follows:

   TOKEN="Bearer $(oc whoami -t)"
   HOST="$(kubectl -n openshift-monitoring get route thanos-querier -o jsonpath='https://{.status.ingress[].host}')"
   echo "TOKEN=$TOKEN" > config/observability/openshift/grafana/datasource.env
   echo "HOST=$HOST" >> config/observability/openshift/grafana/datasource.env
   kubectl apply -k config/observability/openshift/grafana

   Copy to Clipboard Toggle word wrap

4. Configure the example Grafana dashboards as follows:

   kubectl apply -k https://github.com/Kuadrant/kuadrant-operator/examples/dashboards?ref=v1.1.0

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable observability for Kuadrant and any Gateways, set enable: true in the observability section in your Kuadrant custom resource as follows:

   apiVersion: kuadrant.io/v1beta1
   kind: Kuadrant
   metadata:
     name: kuadrant-sample
   spec:
     observability:
       enable: true

   Copy to Clipboard Toggle word wrap

   When enabled, Connectivity Link creates ServiceMonitors and PodMonitors for its components in the Connectivity Link Operator namespace. A single set of monitors is also created in each Gateway namespace to scrape metrics from any Gateways, and in the corresponding Gateway system namespace, in this case, the istio-system namespace.

2. Check the created monitors as follows:

   kubectl get servicemonitor,podmonitor -A -l kuadrant.io/observability=true

   Copy to Clipboard Toggle word wrap
   Note

   You can make changes to monitors after they are created if needed. Monitors are only ever created or deleted, and not updated or reverted. If you decide the default monitors are not suitable, you can set enable: false and create your own ServiceMonitor or PodMonitor definitions, or configure Prometheus directly.

Procedure

1. Enable tracing in OpenShift Service Mesh by configuring your Telemetry custom resource as follows:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: gateway-system
   spec:
     tracing:
     - providers:
       - name: tempo-otlp
       randomSamplingPercentage: 100

   Copy to Clipboard Toggle word wrap

2. Configure a tracing extensionProvider in OpenShift Service Mesh in your Istio custom resource as follows:

   apiVersion: operator.istio.io/v1alpha1
   kind: Istio
   metadata:
     name: default
   spec:
     namespace: gateway-system
     values:
       meshConfig:
         defaultConfig:
           tracing: {}
         enableTracing: true
         extensionProviders:
         - name: tempo-otlp
           opentelemetry:
             port: 4317
             service: tempo.tempo.svc.cluster.local

   Copy to Clipboard Toggle word wrap
   Important

   You must set the OpenTelemetry collector protocol in the service port name or appProtocol fields as described in the OpenShift Service Mesh documentation. For example, when using gRPC, the port name should begin with grpc- or the appProtocol should be grpc.

3. Enable request tracing in your Authorino custom resource and send authentication and authorization traces to the central collector as follows:

   apiVersion: operator.authorino.kuadrant.io/v1beta1
   kind: Authorino
   metadata:
     name: authorino
   spec:
     tracing:
       endpoint: rpc://tempo.tempo.svc.cluster.local:4317

   Copy to Clipboard Toggle word wrap

4. Enable request tracing in your Limitador custom resource and send rate limit traces to the central collector as follows:

   apiVersion: limitador.kuadrant.io/v1alpha1
   kind: Limitador
   metadata:
     name: limitador
   spec:
     tracing:
       endpoint: rpc://tempo.tempo.svc.cluster.local:4317

   Copy to Clipboard Toggle word wrap
   Note

   Ensure that the tracing collector is the same one that OpenShift Service Mesh is sending traces to so that they can all be correlated later.

   When the changes are applied, the Authorino and Limitador components are redeployed with tracing enabled.

   Important

   Trace IDs do not propagate to Wasm modules in OpenShift Service Mesh currently. This means that requests passed to Limitador do not have the relevant parent trace ID. However, if the trace initiation point is outside Service Mesh, the parent trace ID is available to Limitador and included in traces. This impacts on correlating traces from Limitador with traces from Authorino, the Gateway, and any other components in the request path.