Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Metrics and Service Mesh

2.1. Using metrics
Copy link

You can use the OpenShift Container Platform monitoring stack and Red Hat OpenShift Service Mesh to track the health and performance of your applications. You can learn how to monitor metrics and alerts for both standard and ambient mesh modes.

2.1.1. About metrics
Copy link

You can monitor service mesh application health and performance by using the platform monitoring stack to track Layer 4 (L4) and Layer 7 (L7) metrics across sidecar, ztunnel, and waypoint proxies.

Every OpenShift Container Platform installation deploys monitoring stack components by default, and the Cluster Monitoring Operator (CMO) manages them. These components include Prometheus, Alertmanager, Thanos Querier, and others. The CMO also deploys the Telemeter Client, which sends a subset of data from platform Prometheus instances to Red Hat to ease Remote Health Monitoring for clusters.

When you have added your application to the mesh, you can monitor the in-cluster health and performance of your applications running on OpenShift Container Platform with metrics and customized alerts for CPU and memory usage, network connectivity, and other resource usage.

When you have added your application to the mesh in ambient mode, you can monitor the Istio standard metrics of your application from the ztunnel resource and the waypoint proxies. The ztunnel also exposes a variety of DNS and debugging metrics.

Ambient mode uses two proxy layers, which results in two types of metrics for each application service. You can collect L4 TCP metrics from both the ztunnel and the waypoint proxies. You can collect L7 metrics, such as HTTP traffic metrics, from the waypoint proxies.

You can integrate Red Hat OpenShift Service Mesh with user-workload monitoring to enable observability in your service mesh. User-workload monitoring provides access to essential built-in tools. Kiali requires this feature to run the dedicated console for Istio.

Prerequisites

  • You have installed the Red Hat OpenShift Service Mesh Operator.

  • You have enabled the user-workload monitoring.

    Note

    You can enable user-workload monitoring by applying the ConfigMap change for metrics integration. For more information, see "Configuring user workload monitoring".

Procedure

  1. Create a Telemetry resource in the Istio control plane namespace to ensure that Prometheus is a metrics provider, similar to the following example: 

    apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-prometheus-metrics
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus

  2. Create a ServiceMonitor resource that monitors the Istio control plane, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s

  3. Create a PodMonitor resource that collects metrics from the Istio proxies, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: istio-system
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id

    where:

    istio-system
    Specifies that you must apply the PodMonitor object in all mesh namespaces, including the Istio control plane namespace, because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects.
    mesh_id
    Specify the actual mesh ID.
    \\d+
    The additional backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \\d+ with \d+.
    \$
    The backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \$ with $.

  4. To validate that the ServiceMonitor and PodMonitor resources are monitoring the Istio control plane, go to the OpenShift Console, navigate to Observe Metrics, and run the query istio_requests_total. Confirm that the metrics for the Istio request are displayed.

    Note

    The Metrics implementation can take a few minutes for the query to return results.

You can integrate Red Hat OpenShift Service Mesh with user-workload monitoring to enable observability in your service mesh ambient mode. User-workload monitoring provides access to essential built-in tools. Kiali requires this feature to run the dedicated console for Istio.

Prerequisites

  • You have installed the Red Hat OpenShift Service Mesh Operator.

  • You have enabled the user-workload monitoring.

    Note

    You can enable user workload monitoring by applying the ConfigMap change for metrics integration. For more information, see "Configuring user workload monitoring".

Procedure

  1. Create a Telemetry resource in the Istio control plane namespace to ensure that Prometheus is a metrics provider, similar to the following example: 

    apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-prometheus-metrics
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus

  2. Create a ServiceMonitor resource that monitors the Istio control plane, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s

  3. Create a PodMonitor resource in the ztunnel namespace for collecting the ztunnel metrics, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-ztunnel-monitor
  namespace: ztunnel
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id

    where:

    mesh_id
    Specify the actual mesh ID.
    \\d+
    The additional backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \\d+ with \d+.
    \$
    The backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \$ with $.

  4. Optional: Deploy a waypoint proxy to enable the Layer 7 (L7) OpenShift Service Mesh features in ambient mode:

    1. Deploy a waypoint proxy for the bookinfo namespace, similar to the following example: 

      apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: service
  name: waypoint
  namespace: bookinfo
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

    2. Enroll the namespace to use the waypoint by running the following command: 

      $ oc label namespace bookinfo istio.io/use-waypoint=waypoint

    3. Create a PodMonitor resource for collecting waypoint proxies metrics in an application namespace such as bookinfo, similar to the following example: 

      apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-waypoint-monitor
  namespace: bookinfo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id

      where:

      mesh_id
      Specify the actual mesh ID.
      \\d+
      The additional backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \\d+ with \d+.
      \$
      The backslash is only used when you apply this replacement from a command line through heredoc. If you apply this from a YAML file, replace \$ with $.
    Note

    A waypoint proxy generates Layer 4 (L4) and L7 metrics. It scopes these statistics by Envoy proxy functions. The Envoy proxy documentation describes the statistic functions, for example, Upstream connection, Listener, HTTP Connection Manager, TCP proxy, and Router.

2.1.3.1. Verifying metrics in ambient mode
Copy link

You can verify that the metrics for your application available in the OpenShift Console.

Prerequisites

  • You have deployed the Bookinfo application in ambient mode to use the following example. For more information, see "Deploying the Bookinfo application in Istio ambient mode".

Procedure

  1. On the OpenShift Console go to Observe Targets.

  2. Find the status of Metrics Targets by searching for targets such as istiod-monitor, istio-ztunnel-monitor, and istio-waypoint-monitor. You can create istio-waypoint-monitor only if you created the waypoint to use Layer 7 (L7) OpenShift Service Mesh features.

    Note

    The ServiceMonitor resource configuration can take a few minutes to show in the Metrics Targets results.

  3. Send some traffic to the Bookinfo productpage service for generating metrics, by running the following command: 

    $ curl "http://${GATEWAY_URL}/productpage" | grep "<title>"
  4. On the OpenShift Console go to Observe Metrics and run a query such as, istio_build, istio_tcp_received_bytes_total, or istio_requests_total.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top