Observability

Procedure

1. Create a YAML file named servicemonitor.yml to monitor the Istio control plane:

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: istiod-monitor
     namespace: istio-system
   spec:
     targetLabels:
     - app
     selector:
       matchLabels:
         istio: pilot
     endpoints:
     - port: http-monitoring
       interval: 30s

2. Apply the YAML file by running the following command:

   $ oc apply -f servicemonitor.yml

3. Create a YAML file named podmonitor.yml to collect metrics from the Istio proxies:

   Example PodMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: istio-proxies-monitor
     namespace: istio-system 1
   spec:
     selector:
       matchExpressions:
       - key: istio-prometheus-ignore
         operator: DoesNotExist
     podMetricsEndpoints:
     - path: /stats/prometheus
       interval: 30s
       relabelings:
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_container_name"]
         regex: "istio-proxy"
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
       - action: replace
         regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
         replacement: '[$2]:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: replace
         regex: (\d+);((([0-9]+?)(\.|$)){4})
         replacement: '$2:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
         separator: ";"
         targetLabel: "app"
         action: replace
         regex: "(.+);.*|.*;(.+)"
         replacement: "${1}${2}"
       - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
         separator: ";"
         targetLabel: "version"
         action: replace
         regex: "(.+);.*|.*;(.+)"
         replacement: "${1}${2}"
       - sourceLabels: ["__meta_kubernetes_namespace"]
         action: replace
         targetLabel: namespace
       - action: replace
         replacement: "the-mesh-identification-string" 2
         targetLabel: mesh_id

   1

   Specifies that the PodMonitor object must be applied in all mesh namespaces, including the Istio control plane namespace, because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects.

   2

   Specify the actual mesh ID.

4. Apply the YAML file by running the following command:

   $ oc apply -f podmonitor.yml

5. On the OpenShift Console go to Observe → Metrics, and run the query istio_requests_total.

   Note

   The Metrics implementation can take a few minutes for the query to return results.

Procedure

1. Navigate to the Red Hat OpenShift distributed tracing data collection Operator and install the OpenTelemetryCollector resource in the istio-system namespace:

   Example OpenTelemetry Collector in istio-system namespace

   kind: OpenTelemetryCollector
   apiVersion: opentelemetry.io/v1beta1
   metadata:
     name: otel
     namespace: istio-system
   spec:
     observability:
       metrics: {}
     deploymentUpdateStrategy: {}
     config:
       exporters:
         otlp:
           endpoint: 'tempo-sample-distributor.tempo.svc.cluster.local:4317'
           tls:
             insecure: true
       receivers:
         otlp:
           protocols:
             grpc:
               endpoint: '0.0.0.0:4317'
             http: {}
       service:
         pipelines:
           traces:
             exporters:
               - otlp
             receivers:
               - otlp

2. Configure Red Hat OpenShift Service Mesh to enable tracing, and define the distributed tracing data collection tracing providers in your meshConfig:

   Example enabling tracing and defining tracing providers

   apiVersion: sailoperator.io/v1
   kind: Istio
   metadata:
   #  ...
     name: default
   spec:
     namespace: istio-system
   #  ...
     values:
       meshConfig:
         enableTracing: true
         extensionProviders:
         - name: otel
           opentelemetry:
             port: 4317
             service: otel-collector.istio-system.svc.cluster.local 1

   1

   The service field is the OpenTelemetry collector service in the istio-system namespace.

3. Create an Istio Telemetry resource to enable tracers defined in spec.values.meshConfig.ExtensionProviders:

   Example Istio Telemetry resource

   apiVersion: telemetry.istio.io/v1
   kind: Telemetry
   metadata:
     name: otel-demo
     namespace: istio-system
   spec:
     tracing:
       - providers:
           - name: otel
         randomSamplingPercentage: 100

   Note

   Once you verify that you can see traces, lower the randomSamplingPercentage value or set it to default to reduce the number of requests.

4. Create the bookinfo namespace by running the following command:

   $ oc create ns bookinfo

5. Depending on the update strategy you are using, enable sidecar injection in the namespace by running the appropriate commands:

   1. If you are using the InPlace update strategy, run the following command:

      $ oc label namespace curl istio-injection=enabled

   2. If you are using the RevisionBased update strategy, run the following commands:

      1. Display the revision name by running the following command:

         $ oc get istiorevisions.sailoperator.io

         Example output

         NAME      TYPE    READY   STATUS    IN USE   VERSION   AGE
         default   Local   True    Healthy   True     v1.24.3   3m33s

      2. Label the namespace with the revision name to enable sidecar injection by running the following command:

         $ oc label namespace curl istio.io/rev=default

6. Deploy the bookinfo application in the bookinfo namespace by running the following command:

   $ oc apply -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo

7. Generate traffic to the productpage pod to generate traces:

   $ oc exec -it -n bookinfo deployments/productpage-v1 -c istio-proxy -- curl localhost:9080/productpage

8. Validate the integration by running the following command to see traces in the UI:

   $ oc get routes -n tempo tempo-sample-query-frontend

   Note

   The OpenShift route for Jaeger UI must be created in the Tempo namespace. You can either manually create it for the tempo-sample-query-frontend service, or update the Tempo custom resource with .spec.template.queryFrontend.jaegerQuery.ingress.type: route.

Procedure

1. Log in to the Red Hat OpenShift Service Mesh web console.
2. Navigate to Operators → OperatorHub.
3. Type Kiali into the filter box to find the Kiali Operator provided by Red Hat.
4. Click Kiali Operator provided by Red Hat to display information about the Operator.
5. Click Install.
6. On the Operator Installation page, select the stable Update Channel.
7. Select All namespaces on the cluster (default). This installs the Operator in the default openshift-operators project and makes the Operator available to all projects in the cluster.

8. Select the Automatic Approval Strategy.

   Note

   The Manual approval strategy requires a user with appropriate credentials to approve the Operator installation and subscription process.

9. Click Install.
10. The Installed Operators page displays the Kiali Operator’s installation progress.

Procedure

1. Create a ClusterRoleBinding resource for Kiali:

   Example ClusterRoleBinding configuration

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kiali-monitoring-rbac
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: cluster-monitoring-view
   subjects:
   - kind: ServiceAccount
     name: kiali-service-account
     namespace: istio-system

2. Create a Kiali resource and point it to your Istio instance:

   Example Kiali resource configuration

   apiVersion: kiali.io/v1alpha1
   kind: Kiali
   metadata:
     name: kiali-user-workload-monitoring
     namespace: istio-system
   spec:
     external_services:
       prometheus:
         auth:
           type: bearer
           use_kiali_token: true
         thanos_proxy:
           enabled: true
         url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091

3. When the Kiali resource is ready, get the Kiali URL from the Route by running the following command:

   $ echo "https://$(oc get routes -n istio-system kiali -o jsonpath='{.spec.host}')"

4. Follow the URL to open Kiali in your web browser.
5. Navigate to the Traffic Graph tab to check the traffic in the Kiali UI.