Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Metrics and Service Mesh

2.1. Using metrics
Copy link

Monitoring stack components are deployed by default in every OpenShift Container Platform installation and are managed by the Cluster Monitoring Operator (CMO). These components include Prometheus, Alertmanager, Thanos Querier, and others. The CMO also deploys the Telemeter Client, which sends a subset of data from platform Prometheus instances to Red Hat to facilitate Remote Health Monitoring for clusters.

When you have added your application to the mesh, you can monitor the in-cluster health and performance of your applications running on OpenShift Container Platform with metrics and customized alerts for CPU and memory usage, network connectivity, and other resource usage.

When you have added your application to the mesh in ambient mode, you can monitor the Istio standard metrics of your application from the ztunnel resource and the waypoint proxies. The ztunnel also exposes a variety of DNS and debugging metrics.

Ambient mode uses two proxy layers, which results in two types of metrics for each application service. You can collect Layer 4 (L4) TCP metrics from both the ztunnel and the waypoint proxies. You can collect Layer 7 (L7) metrics, such as HTTP traffic metrics, from the waypoint proxies.

You can integrate Red Hat OpenShift Service Mesh with user-workload monitoring to enable observability in your service mesh. User-workload monitoring provides access to essential built-in tools and is required to run Kiali, the dedicated console for Istio.

Prerequisites

  • You have installed the Red Hat OpenShift Service Mesh Operator.

  • You have enabled the user-workload monitoring.

    Note

    You can enable user-workload monitoring by applying the ConfigMap change for metrics integration. For more information, see Configuring user workload monitoring.

Procedure

  1. Create a Telemetry resource in the Istio control plane namespace to ensure that Prometheus is a metrics provider, similar to the following example: 

    apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-prometheus-metrics
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    Copy to Clipboard Toggle word wrap

  2. Create a ServiceMonitor resource that monitors the Istio control plane, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s
    Copy to Clipboard Toggle word wrap

  3. Create a PodMonitor resource that collects metrics from the Istio proxies, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxies-monitor
  namespace: istio-system
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id
    Copy to Clipboard Toggle word wrap

    where:

    istio-system
    Specifies that the PodMonitor object must be applied in all mesh namespaces, including the Istio control plane namespace, because OpenShift Container Platform monitoring ignores the namespaceSelector spec in ServiceMonitor and PodMonitor objects.
    mesh_id
    Specify the actual mesh ID.
    \\d+
    The additional backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \\d+ with \d+.
    \$
    The backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \$ with $.

  4. To validate that the ServiceMonitor and PodMonitor resources are monitoring the Istio control plane, go to the OpenShift Console, navigate to Observe Metrics, and run the query istio_requests_total. Confirm that the metrics for the Istio request are displayed.

    Note

    The Metrics implementation can take a few minutes for the query to return results.

You can integrate Red Hat OpenShift Service Mesh with user-workload monitoring to enable observability in your service mesh ambient mode. User-workload monitoring provides access to essential built-in tools and is required to run Kiali, the dedicated console for Istio.

Prerequisites

  • You have installed the Red Hat OpenShift Service Mesh Operator.

  • You have enabled the user-workload monitoring.

    Note

    You can enable user workload monitoring by applying the ConfigMap change for metrics integration. For more information, see Configuring user workload monitoring.

Procedure

  1. Create a Telemetry resource in the Istio control plane namespace to ensure that Prometheus is a metrics provider, similar to the following example: 

    apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-prometheus-metrics
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    Copy to Clipboard Toggle word wrap

  2. Create a ServiceMonitor resource that monitors the Istio control plane, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod-monitor
  namespace: istio-system
spec:
  targetLabels:
  - app
  selector:
    matchLabels:
      istio: pilot
  endpoints:
  - port: http-monitoring
    interval: 30s
    Copy to Clipboard Toggle word wrap

  3. Create a PodMonitor resource in the ztunnel namespace for collecting the ztunnel metrics, similar to the following example: 

    apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-ztunnel-monitor
  namespace: ztunnel
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id
    Copy to Clipboard Toggle word wrap

    where:

    mesh_id
    Specify the actual mesh ID.
    \\d+
    The additional backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \\d+ with \d+.
    \$
    The backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \$ with $.

  4. Optional: Deploy a waypoint proxy to enable the Layer 7 (L7) OpenShift Service Mesh features in ambient mode:

    1. Deploy a waypoint proxy for the bookinfo namespace, similar to the following example: 

      apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: service
  name: waypoint
  namespace: bookinfo
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
      Copy to Clipboard Toggle word wrap

    2. Enroll the namespace to use the waypoint by running the following command: 

      $ oc label namespace bookinfo istio.io/use-waypoint=waypoint
      Copy to Clipboard Toggle word wrap

    3. Create a PodMonitor resource for collecting waypoint proxies metrics in an application namespace such as bookinfo, similar to the following example: 

      apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-waypoint-monitor
  namespace: bookinfo
spec:
  selector:
    matchExpressions:
    - key: istio-prometheus-ignore
      operator: DoesNotExist
  podMetricsEndpoints:
  - path: /stats/prometheus
    interval: 30s
    relabelings:
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_container_name]
      regex: "istio-proxy"
    - action: keep
      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
    - action: replace
      regex: (\\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
      replacement: '[\$2]:\$1'
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    - action: replace
      regex: (\\d+);((([0-9]+?)(\.|$)){4})
      replacement: \$2:\$1
      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]
      targetLabel: __address__
    # Set the 'app' label from 'app.kubernetes.io/name' or fallback to 'app'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app"]
      separator: ";"
      targetLabel: "app"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # Set the 'version' label from 'app.kubernetes.io/version' or fallback to 'version'
    - sourceLabels: ["__meta_kubernetes_pod_label_app_kubernetes_io_version", "__meta_kubernetes_pod_label_version"]
      separator: ";"
      targetLabel: "version"
      action: replace
      regex: "(.+);.*|.*;(.+)"
      replacement: "\${1}\${2}"  # Use the first non-empty value
    # additional labels
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - action: replace
      replacement: "mesh_id"
      targetLabel: mesh_id
      Copy to Clipboard Toggle word wrap

      where:

      mesh_id
      Specify the actual mesh ID.
      \\d+
      The additional backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \\d+ with \d+.
      \$
      The backslash is only used when you apply this replacement from a command line via heredoc. If you apply this from a yaml file, replace \$ with $.
    Note

    A waypoint proxy generates Layer 4 (L4) and L7 metrics. It scopes these statistics by Envoy proxy functions. The Envoy statistic functions are described in Envoy proxy documentation, for example, Upstream connection, Listener, HTTP Connection Manager, TCP proxy, and Router.

2.1.2.1. Verifying metrics in ambient mode
Copy link

You can verify that the metrics for your application available in the OpenShift Console.

Prerequisites

Procedure

  1. On the OpenShift Console go to Observe Targets.

  2. Find the status of Metrics Targets by searching for targets such as istiod-monitor, istio-ztunnel-monitor, and istio-waypoint-monitor. istio-waypoint-monitor is created only when the waypoint proxy is created to use Layer 7 (L7) OpenShift Service Mesh features.

    Note

    The ServiceMonitor resource configuration can take a few minutes to show in the Metrics Targets results.

  3. Send some traffic to the Bookinfo productpage service for generating metrics, by running the following command: 

    $ curl "http://${GATEWAY_URL}/productpage" | grep "<title>"
    Copy to Clipboard Toggle word wrap
  4. On the OpenShift Console go to Observe Metrics and run a query such as, istio_build, istio_tcp_received_bytes_total, or istio_requests_total.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top