Chapter 2. Metrics and Service Mesh

Procedure

1. Apply a ServiceMonitor object to monitor the Istio control plane:

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: istiod-monitor
     namespace: istio-system
   spec:
     targetLabels:
     - app
     selector:
       matchLabels:
         istio: pilot
     endpoints:
     - port: http-monitoring
       interval: 30s

2. Apply a PodMonitor object to collect metrics from Istio proxies:

   Example PodMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: istio-proxies-monitor
     namespace: istio-system 1
   spec:
     selector:
       matchExpressions:
       - key: istio-prometheus-ignore
         operator: DoesNotExist
     podMetricsEndpoints:
     - path: /stats/prometheus
       interval: 30s
       relabelings:
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_container_name"]
         regex: "istio-proxy"
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
       - action: replace
         regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
         replacement: '[$2]:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: replace
         regex: (\d+);((([0-9]+?)(\.|$)){4})
         replacement: '$2:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: labeldrop
         regex: "__meta_kubernetes_pod_label_(.+)"
       - sourceLabels: ["__meta_kubernetes_namespace"]
         action: replace
         targetLabel: namespace
       - sourceLabels: ["__meta_kubernetes_pod_name"]
         action: replace
         targetLabel: pod_name

   1

   Because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects, you must apply the PodMonitor object in all mesh namespaces, including the Istio control plane namespace.

3. Go to the OpenShift Console Observe Metrics, and run the query istio_requests_total.