Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 4. Customizing observability configuration

Customize the observability configuration to the specific needs of your environment, after you enable observability.

To learn more about how you want to manage and view cluster fleet data that the observability service collects, read the following sections:

Required access: Cluster administrator

4.1. Creating custom rules
Copiar enlace

Create custom rules for the observability installation by adding Prometheus recording rules and alerting rules to the observability resource.

To precalculate expensive expressions, use the recording rules abilities. The results are saved as a new set of time series. With the alerting rules, you can specify the alert conditions based on how you want to send an alert to an external service.

Note: When you update your custom rules, observability-thanos-rule pods restart automatically.

Define custom rules with Prometheus to create alert conditions, and send notifications to an external messaging service. View the following examples of custom rules:

  • Create a custom alert rule. Create a config map named thanos-ruler-custom-rules in the open-cluster-management-observability namespace. You must name the key, custom_rules.yaml, as shown in the following example. You can create multiple rules in the configuration.

    • Create a custom alert rule that notifies you when your CPU usage passes your defined value. Your YAML might resemble the following content: 
    data:
  custom_rules.yaml: |
    groups:
      - name: cluster-health
        rules:
        - alert: ClusterCPUHealth-jb
          annotations:
            summary: Notify when CPU utilization on a cluster is greater than the defined utilization limit
            description: "The cluster has a high CPU usage: {{ $value }} core for {{ $labels.cluster }} {{ $labels.clusterID }}."
          expr: |
            max(cluster:cpu_usage_cores:sum) by (clusterID, cluster, prometheus) > 0
          for: 5s
          labels:
            cluster: "{{ $labels.cluster }}"
            prometheus: "{{ $labels.prometheus }}"
            severity: critical
    Copy to Clipboard Toggle word wrap

    +

    • The default alert rules are in the thanos-ruler-default-rules config map of the open-cluster-management-observability namespace.

  • Create a custom recording rule within the thanos-ruler-custom-rules config map. Create a recording rule that gives you the ability to get the sum of the container memory cache of a pod. Your YAML might resemble the following content: 

    data:
  custom_rules.yaml: |
    groups:
      - name: container-memory
        rules:
        - record: pod:container_memory_cache:sum
          expr: sum(container_memory_cache{pod!=""}) BY (pod, container)
    Copy to Clipboard Toggle word wrap

    Note: After you make changes to the config map, the configuration automatically reloads. The configuration reloads because of the config-reload within the observability-thanos-ruler sidecar.

  • To verify that the alert rules are functioning correctly, go to the Grafana dashboard, select to the Explore page, and query ALERTS. The alert is only available in Grafana if you created the alert.

4.2. Adding custom metrics
Copiar enlace

Add metrics to the metrics_list.yaml file, to collect from managed clusters. Complete the following steps:

  1. Before you add a custom metric, verify that mco observability is enabled with the following command: 

    oc get mco observability -o yaml
    Copy to Clipboard Toggle word wrap

  2. Check for the following message in the status.conditions.message section reads: 

    Observability components are deployed and running
    Copy to Clipboard Toggle word wrap

  3. Create the observability-metrics-custom-allowlist config map in the open-cluster-management-observability namespace with the following command: 

    oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml
    Copy to Clipboard Toggle word wrap

  4. Add the name of the custom metric to the metrics_list.yaml parameter. Your YAML for the config map might resemble the following content: 

    kind: ConfigMap
apiVersion: v1
metadata:
  name: observability-metrics-custom-allowlist
data:
  metrics_list.yaml: |
    names:
    1 
    
      - node_memory_MemTotal_bytes
    rules:
    2 
    
    - record: apiserver_request_duration_seconds:histogram_quantile_90
      expr: histogram_quantile(0.90,sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\",
        verb!=\"WATCH\"}[5m])) by (verb,le))
    Copy to Clipboard Toggle word wrap
    1
    Optional: Add the name of the custom metrics that are to be collected from the managed cluster.
    2
    Optional: Enter only one value for the expr and record parameter pair to define the query expression. The metrics are collected as the name that is defined in the record parameter from your managed cluster. The metric value returned are the results after you run the query expression.

    You can use either one or both of the sections. For user workload metrics, see the Adding user workload metrics section.

  5. Verify the data collection from your custom metric by querying the metric from the Explore page. You can also use the custom metrics in your own dashboard.

4.2.1. Adding user workload metrics
Copiar enlace

Collect OpenShift Container Platform user-defined metrics from workloads in OpenShift Container Platform to display the metrics from your Grafana dashboard. Complete the following steps:

  1. Enable monitoring on your OpenShift Container Platform cluster. See Enabling monitoring for user-defined projects in the Additional resources section.

    If you have a managed cluster with monitoring for user-defined workloads enabled, the user workloads are located in the test namespace and generate metrics. These metrics are collected by Prometheus from the OpenShift Container Platform user workload.

  2. Add user workload metrics to the observability-metrics-custom-allowlist config map to collect the metrics in the test namespace. View the following example: 

    kind: ConfigMap
apiVersion: v1
metadata:
  name: observability-metrics-custom-allowlist
  namespace: test
data:
  uwl_metrics_list.yaml:
    1 
    
    names:
    2 
    
      - sample_metrics
    Copy to Clipboard Toggle word wrap
    1
    Enter the key for the config map data.
    2
    Enter the value of the config map data in YAML format. The names section includes the list of metric names, which you want to collect from the test namespace. After you create the config map, the observability collector collects and pushes the metrics from the target namespace to the hub cluster.

4.2.2. Removing default metrics
Copiar enlace

If you do not want to collect data for a specific metric from your managed cluster, remove the metric from the observability-metrics-custom-allowlist.yaml file. When you remove a metric, the metric data is not collected from your managed clusters. Complete the following steps to remove a default metric:

  1. Verify that mco observability is enabled by using the following command: 

    oc get mco observability -o yaml
    Copy to Clipboard Toggle word wrap

  2. Add the name of the default metric to the metrics_list.yaml parameter with a hyphen - at the start of the metric name. View the following metric example: 

    -cluster_infrastructure_provider
    Copy to Clipboard Toggle word wrap

  3. Create the observability-metrics-custom-allowlist config map in the open-cluster-management-observability namespace with the following command: 

    oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml
    Copy to Clipboard Toggle word wrap
  4. Verify that the observability service is not collecting the specific metric from your managed clusters. When you query the metric from the Grafana dashboard, the metric is not displayed.

4.3. Adding advanced configuration for retention
Copiar enlace

Add the advanced configuration section to update the retention for each observability component, according to your needs.

Edit the MultiClusterObservability custom resource and add the advanced section with the following command: 

oc edit mco observability -o yaml
Copy to Clipboard Toggle word wrap

Your YAML file might resemble the following contents: 

spec:
  advanced:
    retentionConfig:
      blockDuration: 2h
      deleteDelay: 48h
      retentionInLocal: 24h
      retentionResolutionRaw: 30d
      retentionResolution5m: 180d
      retentionResolution1h: 0d
    receive:
      resources:
        limits:
          memory: 4096Gi
      replicas: 3
Copy to Clipboard Toggle word wrap

For descriptions of all the parameters that can added into the advanced configuration, see the Observability API documentation.

4.4. Dynamic metrics for single-node OpenShift clusters
Copiar enlace

Dynamic metrics collection supports automatic metric collection based on certain conditions. By default, a single-node OpenShift cluster does not collect pod and container resource metrics. Once a single-node OpenShift cluster reaches a specific level of resource consumption, the defined granular metrics are collected dynamically. When the cluster resource consumption is consistently less than the threshold for a period of time, granular metric collection stops.

The metrics are collected dynamically based on the conditions on the managed cluster specified by a collection rule. Because these metrics are collected dynamically, the following Red Hat Advanced Cluster Management Grafana dashboards do not display any data. When a collection rule is activated and the corresponding metrics are collected, the following panels display data for the duration of the time that the collection rule is initiated:

  • Kubernetes/Compute Resources/Namespace (Pods)
  • Kubernetes/Compute Resources/Namespace (Workloads)
  • Kubernetes/Compute Resources/Nodes (Pods)
  • Kubernetes/Compute Resources/Pod
  • Kubernetes/Compute Resources/Workload A collection rule includes the following conditions:
  • A set of metrics to collect dynamically.
  • Conditions written as a PromQL expression.
  • A time interval for the collection, which must be set to true.
  • A match expression to select clusters where the collect rule must be evaluated.

By default, collection rules are evaluated continuously on managed clusters every 30 seconds, or at a specific time interval. The lowest value between the collection interval and time interval takes precedence. Once the collection rule condition persists for the duration specified by the for attribute, the collection rule starts and the metrics specified by the rule are automatically collected on the managed cluster. Metrics collection stops automatically after the collection rule condition no longer exists on the managed cluster, at least 15 minutes after it starts.

The collection rules are grouped together as a parameter section named collect_rules, where it can be enabled or disabled as a group. Red Hat Advanced Cluster Management installation includes the collection rule group, SNOResourceUsage with two default collection rules: HighCPUUsage and HighMemoryUsage. The HighCPUUsage collection rule begins when the node CPU usage exceeds 70%. The HighMemoryUsage collection rule begins if the overall memory utilization of the single-node OpenShift cluster exceeds 70% of the available node memory. Currently, the previously mentioned thresholds are fixed and cannot be changed. When a collection rule begins for more than the interval specified by the for attribute, the system automatically starts collecting the metrics that are specified in the dynamic_metrics section.

View the list of dynamic metrics that from the collect_rules section, in the following YAML file: 

collect_rules:
  - group: SNOResourceUsage
    annotations:
      description: >
        By default, a {sno} cluster does not collect pod and container resource metrics. Once a {sno} cluster
        reaches a level of resource consumption, these granular metrics are collected dynamically.
        When the cluster resource consumption is consistently less than the threshold for a period of time,
        collection of the granular metrics stops.
    selector:
      matchExpressions:
        - key: clusterType
          operator: In
          values: ["{sno}"]
    rules:
    - collect: SNOHighCPUUsage
      annotations:
        description: >
          Collects the dynamic metrics specified if the cluster cpu usage is constantly more than 70% for 2 minutes
      expr: (1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100 > 70
      for: 2m
      dynamic_metrics:
        names:
          - container_cpu_cfs_periods_total
          - container_cpu_cfs_throttled_periods_total
          - kube_pod_container_resource_limits
          - kube_pod_container_resource_requests
          - namespace_workload_pod:kube_pod_owner:relabel
          - node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
          - node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
    - collect: SNOHighMemoryUsage
      annotations:
        description: >
          Collects the dynamic metrics specified if the cluster memory usage is constantly more than 70% for 2 minutes
      expr: (1 - sum(:node_memory_MemAvailable_bytes:sum) / sum(kube_node_status_allocatable{resource=\"memory\"})) * 100 > 70
      for: 2m
      dynamic_metrics:
        names:
          - kube_pod_container_resource_limits
          - kube_pod_container_resource_requests
          - namespace_workload_pod:kube_pod_owner:relabel
        matches:
          - __name__="container_memory_cache",container!=""
          - __name__="container_memory_rss",container!=""
          - __name__="container_memory_swap",container!=""
          - __name__="container_memory_working_set_bytes",container!=""
Copy to Clipboard Toggle word wrap

A collect_rules.group can be disabled in the custom-allowlist as shown in the following example. When a collect_rules.group is disabled, metrics collection reverts to the previous behavior. These metrics are collected at regularly, specified intervals: 

collect_rules:
  - group: -SNOResourceUsage
Copy to Clipboard Toggle word wrap

The data is only displayed in Grafana when the rule is initiated.

If your workload increases, increase the number of replicas of your observability pods. Navigate to the Red Hat OpenShift Container Platform console from your hub cluster. Locate the MultiClusterObservability custom resource, and update the replicas parameter value for the component where you want to change the replicas. Your updated YAML might resemble the following content: 

spec:
   advanced:
      receive:
         replicas: 6
Copy to Clipboard Toggle word wrap

For more information about the parameters within the mco observability custom resource, see the Observability API documentation.

4.6. Customizing route certificate
Copiar enlace

If you want to customize the OpenShift Container Platform route certification, you must add the routes in the alt_names section. To ensure your OpenShift Container Platform routes are accessible, add the following information: alertmanager.apps.<domainname>, observatorium-api.apps.<domainname>, rbac-query-proxy.apps.<domainname>.

For more details, see Replacing certificates for alertmanager route in the Governance documentation.

Note: Users are responsible for certificate rotations and updates.

4.7. Customizing certificates for accessing the object store
Copiar enlace

Complete the following steps to customize certificates for accessing the object store:

  1. Edit the http_config section by adding the certificate in the object store secret. View the following example: 

     thanos.yaml: |
    type: s3
    config:
      bucket: "thanos"
      endpoint: "minio:9000"
      insecure: false
      access_key: "minio"
      secret_key: "minio123"
      http_config:
        tls_config:
          ca_file: /etc/minio/certs/ca.crt
          insecure_skip_verify: false
    Copy to Clipboard Toggle word wrap

  2. Add the object store secret in the open-cluster-management-observability namespace. The secret must contain the ca.crt that you defined in the previous secret example. If you want to enable Mutual TLS, you need to provide public.crt, and private.key in the previous secret. View the following example: 

     thanos.yaml: |
    type: s3
    config:
      ...
      http_config:
        tls_config:
          ca_file: /etc/minio/certs/ca.crt
    1 
    
          cert_file: /etc/minio/certs/public.crt
          key_file: /etc/minio/certs/private.key
          insecure_skip_verify: false
    Copy to Clipboard Toggle word wrap
    1
    The path to certificates and key values for the thanos-object-storage secret.

  3. Configure the secret name and mount path by updating the tlsSecretName and tlsSecretMountPath parameters in the MultiClusterObservability custom resource. View the following example where the secret name is tls-certs-secret and the mount path for the certificates and key value is the directory that is used in the prior example: 

    metricObjectStorage:
      key: thanos.yaml
      name: thanos-object-storage
      tlsSecretName: tls-certs-secret
      tlsSecretMountPath: /etc/minio/certs
    Copy to Clipboard Toggle word wrap

  4. Mount the secret in the tlsSecretMountPath resource of all components that need to access the object store by renaming the existing TLS. See the following example: 

    metricObjectStorage:
      key: thanos.yaml
      name: thanos-object-storage
      tlsSecretName: <existing-tls-certs-secret>
      tlsSecretMountPath: /etc/minio/certs
        receiver:
        store:
        ruler:
        compact:
    Copy to Clipboard Toggle word wrap
  5. To verify that you can access the object store, check that the pods are displayed.

4.8. Configuring proxy settings for observability add-ons
Copiar enlace

Configure the proxy settings to allow the communications from the managed cluster to access the hub cluster through a HTTP and HTTPS proxy server. Typically, add-ons do not need any special configuration to support HTTP and HTTPS proxy servers between a hub cluster and a managed cluster. But if you enabled the observability add-on, you must complete the proxy configuration.

4.9. Prerequisite
Copiar enlace

  • You have a hub cluster.
  • You have enabled the proxy settings between the hub cluster and managed cluster.

Complete the following steps to configure the proxy settings for the observability add-on:

  1. Go to the cluster namespace on your hub cluster.

  2. Create an AddOnDeploymentConfig resource with the proxy settings by adding a spec.proxyConfig parameter. View the following YAML example: 

    apiVersion: addon.open-cluster-management.io/v1alpha1
kind: AddOnDeploymentConfig
metadata:
  name: <addon-deploy-config-name>
  namespace: <managed-cluster-name>
spec:
  agentInstallNamespace: open-cluster-managment-addon-observability
  proxyConfig:
    httpsProxy: "http://<username>:<password>@<ip>:<port>"
    1 
    
    noProxy: ".cluster.local,.svc,172.30.0.1"
    2
    Copy to Clipboard Toggle word wrap
    1
    For this field, you can specify either a HTTP proxy or a HTTPS proxy.
    2
    Include the IP address of the kube-apiserver.

  3. To get the IP address, run following command on your managed cluster: 

    oc -n default describe svc kubernetes | grep IP:
    Copy to Clipboard Toggle word wrap

  4. Go to the ManagedClusterAddOn resource and update it by referencing the AddOnDeploymentConfig resource that you made. View the following YAML example: 

    apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: observability-controller
  namespace: <managed-cluster-name>
spec:
  installNamespace: open-cluster-managment-addon-observability
  configs:
  - group: addon.open-cluster-management.io
    resource: AddonDeploymentConfig
    name: <addon-deploy-config-name>
    namespace: <managed-cluster-name>
    Copy to Clipboard Toggle word wrap

  5. Verify the proxy settings. If you successfully configured the proxy settings, the metric collector deployed by the observability add-on agent on the managed cluster sends the data to the hub cluster. Complete the following steps:

    1. Go to the hub cluster then the managed cluster on the Grafana dashboard.
    2. View the metrics for the proxy settings.

4.10. Disabling proxy settings for observability add-ons
Copiar enlace

If your development needs change, you might need to disable the proxy setting for the observability add-ons you configured for the hub cluster and managed cluster. You can disable the proxy settings for the observability add-on at any time. Complete the following steps:

  1. Go to the ManagedClusterAddOn resource.
  2. Remove the referenced AddOnDeploymentConfig resource.

4.11. Additional resources
Copiar enlace

Volver arriba
Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2025 Red Hat