Chapter 2. Metrics and Service Mesh

Procedure

1. Create a YAML file named servicemonitor.yml to monitor the Istio control plane:

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: istiod-monitor
     namespace: istio-system
   spec:
     targetLabels:
     - app
     selector:
       matchLabels:
         istio: pilot
     endpoints:
     - port: http-monitoring
       interval: 30s

2. Apply the YAML file by running the following command:

   $ oc apply -f servicemonitor.yml

3. Create a YAML file named podmonitor.yml to collect metrics from the Istio proxies:

   Example PodMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: istio-proxies-monitor
     namespace: istio-system 1
   spec:
     selector:
       matchExpressions:
       - key: istio-prometheus-ignore
         operator: DoesNotExist
     podMetricsEndpoints:
     - path: /stats/prometheus
       interval: 30s
       relabelings:
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_container_name"]
         regex: "istio-proxy"
       - action: keep
         sourceLabels: ["__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape"]
       - action: replace
         regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
         replacement: '[$2]:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: replace
         regex: (\d+);((([0-9]+?)(\.|$)){4})
         replacement: '$2:$1'
         sourceLabels: ["__meta_kubernetes_pod_annotation_prometheus_io_port","__meta_kubernetes_pod_ip"]
         targetLabel: "__address__"
       - action: labeldrop
         regex: "__meta_kubernetes_pod_label_(.+)"
       - sourceLabels: ["__meta_kubernetes_namespace"]
         action: replace
         targetLabel: namespace
       - sourceLabels: ["__meta_kubernetes_pod_name"]
         action: replace
         targetLabel: pod_name

   1

   Because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects, you must apply the PodMonitor object in all mesh namespaces, including the Istio control plane namespace.

4. Apply the YAML file by running the following command:

   $ oc apply -f podmonitor.yml

5. On the OpenShift Console go to Observe Metrics, and run the query istio_requests_total.

   Note

   The Metrics implementation can take a few minutes for the query to return results.