Chapter 2. Metrics and Service Mesh

Procedure

1. Create a YAML file named servicemonitor.yml to monitor the Istio control plane:

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: istiod-monitor
     namespace: istio-system
   spec:
     targetLabels:
     - app
     selector:
       matchLabels:
         istio: pilot
     endpoints:
     - port: http-monitoring
       interval: 30s

2. Apply the YAML file by running the following command:

   $ oc apply -f servicemonitor.yml

3. Create a YAML file named podmonitor.yml to collect metrics from the Istio proxies:

   Example PodMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: istio-proxies-monitor
     namespace: istio-system
   spec:
     selector:
       matchExpressions:
       - key: istio-prometheus-ignore
         operator: DoesNotExist
     podMetricsEndpoints:
     - path: /stats/prometheus
       interval: 30s
       relabelings:
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_container_name"]
         regex: "istio-proxy"
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
       - action: replace
         regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
         replacement: '[$2]:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: replace
         regex: (\d+);((([0-9]+?)(\.|$)){4})
         replacement: '$2:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app"]
         separator: ";"
         targetLabel: "app"
         action: replace
         regex: "(.+);.*|.*;(.+)"
         replacement: "${1}${2}"
       - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version","__meta_kubernetes_pod_label_version"]
         separator: ";"
         targetLabel: "version"
         action: replace
         regex: "(.+);.*|.*;(.+)"
         replacement: "${1}${2}"
       - sourceLabels: ["__meta_kubernetes_namespace"]
         action: replace
         targetLabel: namespace
       - action: replace
         replacement: "the-mesh-identification-string"
         targetLabel: mesh_id

   • metadata.namespace specifies that the PodMonitor object must be applied in all mesh namespaces, including the Istio control plane namespace, because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects.
   • spec.podMetricsEndpoints.relabelings.replacement: "the-mesh-identification-string" specifies the actual mesh ID.

4. Apply the YAML file by running the following command:

   $ oc apply -f podmonitor.yml

   istio-system

   Specifies that you must apply the PodMonitor object in all mesh namespaces, including the Istio control plane namespace, because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects.

   mesh_id

   Specify the actual mesh ID.

   \\d+

   The additional backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \\d+ with \d+.

   \$

   The backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \$ with $.

5. To validate that the ServiceMonitor and PodMonitor resources are monitoring the Istio control plane, go to the OpenShift Console, navigate to Observe Metrics, and run the query istio_requests_total. Confirm that the metrics for the Istio request are displayed.

   Note

   The Metrics implementation can take a few minutes for the query to return results.