Chapter 1. Connectivity Link observability

Procedure

1. Verify that user workload monitoring is configured correctly in your OpenShift Container Platform cluster as follows:

   $ kubectl get configmap cluster-monitoring-config -n openshift-monitoring -o jsonpath='{.data.config\.yaml}'|grep enableUserWorkload

   Copy to Clipboard Toggle word wrap

   The expected output is enableUserWorkload: true.

2. Install the Connectivity Link, Gateway, and Grafana component metrics and configuration as follows:

   kubectl apply -k https://github.com/Kuadrant/kuadrant-operator/config/install/configure/observability?ref=v1.1.0

   Copy to Clipboard Toggle word wrap

3. From the root directory of your Kuadrant Operator repository, configure the OpenShift Container Platform thanos-query instance as a data source in Grafana as follows:

   TOKEN="Bearer $(oc whoami -t)"
   HOST="$(kubectl -n openshift-monitoring get route thanos-querier -o jsonpath='https://{.status.ingress[].host}')"
   echo "TOKEN=$TOKEN" > config/observability/openshift/grafana/datasource.env
   echo "HOST=$HOST" >> config/observability/openshift/grafana/datasource.env
   kubectl apply -k config/observability/openshift/grafana

   Copy to Clipboard Toggle word wrap

4. Configure the example Grafana dashboards as follows:

   kubectl apply -k https://github.com/Kuadrant/kuadrant-operator/examples/dashboards?ref=v1.1.0

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable observability for Kuadrant and any gateways, set enable: true in the observability section in your Kuadrant Custom Resource:

   apiVersion: kuadrant.io/v1beta1
   kind: Kuadrant
   metadata:
     name: kuadrant-sample
   spec:
     observability:
       enable: true

   Copy to Clipboard Toggle word wrap

   When enabled, Connectivity Link creates ServiceMonitors and PodMonitors for its components in the Connectivity Link Operator namespace. A single set of monitors is also created in each Gateway namespace to scrape metrics from any gateways, and in the corresponding Gateway system namespace, in this case, the istio-system namespace.

Verification

1. Check the created monitors by running the following command:

   $ kubectl get servicemonitor,podmonitor -A -l kuadrant.io/observability=true

   Copy to Clipboard Toggle word wrap
   Note

   You can delete and re-create monitors as required. Monitors are only ever created or deleted, and not updated or reverted. If you decide the default monitors are not suitable, you can set enable: false and create your own ServiceMonitor or PodMonitor definitions, or configure Prometheus directly.

Procedure

1. Importing dashboards in Grafana::

   1. Click Dashboards > New > Import, and use one of the following options:

      • Upload a dashboard JSON file.
      • Enter a dashboard ID obtained from the Grafana dashboards website.

      • Enter JSON content directly.

        For more information, see the Grafana documentation on how to import dashboards.

1. Importing dashboards automatically in OpenShift Container Platform::

   1. You can automate dashboard provisioning in Grafana by adding JSON files to a ConfigMap, which must be mounted at /etc/grafana/provisioning/dashboards.

      Tip

      Alternatively, to avoid adding ConfigMap volume mounts in your Grafana deployment, you can use a GrafanaDashboard resource to reference a ConfigMap. For an example, see Dashboard from ConfigMap in the Grafana documentation.

      Data sources are configured as template variables, automatically integrating with your existing data sources. The metrics for these dashboards are sourced from Prometheus.

      Note

      For some example dashboard panels to work correctly, HTTPRoutes in Connectivity Link must include a service and deployment label with a value that matches the name of the service and deployment being routed to, for example, service=my-app and deployment=my-app. This allows low-level Istio and Envoy metrics to be joined with Gateway API state metrics.

2. Configuring Prometheus alerts

   1. You can integrate the Kuadrant example alerts into Prometheus as PrometheusRule resources, and then adjust the alert thresholds to suit your specific operational needs.
   2. Service Level Objective (SLO) alerts generated by using the Sloth GitHub project are also included. You can use these alerts to integrate with the SLO Grafana dashboard, which uses generated labels to comprehensively overview your SLOs.
   3. For details on how to configure Prometheus alerts, see the OpenShift documentation on managing alerting rules.

Procedure

1. Enable tracing in OpenShift Service Mesh by configuring your Telemetry custom resource as follows:

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: gateway-system
   spec:
     tracing:
     - providers:
       - name: tempo-otlp
       randomSamplingPercentage: 100

   Copy to Clipboard Toggle word wrap

2. Configure a tracing extensionProvider for OpenShift Service Mesh in your Istio custom resource as follows:

   apiVersion: operator.istio.io/v1alpha1
   kind: Istio
   metadata:
     name: default
   spec:
     namespace: gateway-system
     values:
       meshConfig:
         defaultConfig:
           tracing: {}
         enableTracing: true
         extensionProviders:
         - name: tempo-otlp
           opentelemetry:
             port: 4317
             service: tempo.tempo.svc.cluster.local

   Copy to Clipboard Toggle word wrap
   Important

   You must set the OpenTelemetry collector protocol in the service port name or appProtocol fields as described in the OpenShift Service Mesh documentation. For example, when using gRPC, the port name should begin with grpc- or the appProtocol should be grpc.

3. Enable request tracing in your Authorino Custom Resource (CR) and send authentication and authorization traces to the central collector as follows:

   apiVersion: operator.authorino.kuadrant.io/v1beta1
   kind: Authorino
   metadata:
     name: authorino
   spec:
     tracing:
       endpoint: rpc://tempo.tempo.svc.cluster.local:4317

   Copy to Clipboard Toggle word wrap

4. Enable request tracing in your Limitador CR and send rate limit traces to the central collector as follows:

   apiVersion: limitador.kuadrant.io/v1alpha1
   kind: Limitador
   metadata:
     name: limitador
   spec:
     tracing:
       endpoint: rpc://tempo.tempo.svc.cluster.local:4317

   Copy to Clipboard Toggle word wrap
   Tip

   Ensure that the tracing collector is the same one that OpenShift Service Mesh is sending traces to so that they can all be correlated later.

   When the changes are applied, the Authorino and Limitador components are redeployed with tracing enabled.

   Important

   Trace IDs do not propagate to WASM modules in OpenShift Service Mesh currently. This means that requests passed to Limitador do not have the relevant parent trace ID. However, if the trace initiation point is outside Service Mesh, the parent trace ID is available to Limitador and included in traces. This impacts on correlating traces from Limitador with traces from Authorino, the Gateway, and any other components in the request path.