Chapter 19. Monitoring clusters that run on RHOSO

Procedure

1. Configure your RHOSO management cluster to send metrics to Prometheus:

   1. Create a secret that is named mtls-bundle in the openstack namespace that contains HTTPS client certificates for authentication to Prometheus by entering the following command:

      $ oc --namespace openstack \
          create secret generic mtls-bundle \
              --from-file=./ca.crt \
              --from-file=osp-client.crt \
              --from-file=osp-client.key

   2. Open the controlplane configuration for editing by running the following command:

      $ oc -n openstack edit openstackcontrolplane/controlplane

   3. With the configuration open, replace the .spec.telemetry.template.metricStorage section so that RHOSO sends metrics to Prometheus. As an example:

            metricStorage:
              customMonitoringStack:
                alertmanagerConfig:
                  disabled: false
                logLevel: info
                prometheusConfig:
                  scrapeInterval: 30s
                  remoteWrite:
                  - url: https://external-prometheus.example.com/api/v1/write 1
                    tlsConfig:
                      ca:
                        secret:
                          name: mtls-bundle
                          key: ca.crt
                      cert:
                        secret:
                          name: mtls-bundle
                          key: ocp-client.crt
                      keySecret:
                        name: mtls-bundle
                        key: ocp-client.key
                  replicas: 2
                resourceSelector:
                  matchLabels:
                    service: metricStorage
                resources:
                  limits:
                    cpu: 500m
                    memory: 512Mi
                  requests:
                    cpu: 100m
                    memory: 256Mi
                retention: 1d 2
              dashboardsEnabled: false
              dataplaneNetwork: ctlplane
              enabled: true
              prometheusTls: {}

      1

      Replace this URL with the URL of your Prometheus instance.

      2

      Set a retention period. Optionally, you can reduce retention for local metrics because of external collection.

2. Configure the tenant cluster on which your workloads run to send metrics to Prometheus:

   1. Create a cluster monitoring config map as a YAML file. The map must include a remote write configuration and cluster identifiers. As an example:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          prometheusK8s:
            retention: 1d 1
            remoteWrite:
            - url: "https://external-prometheus.example.com/api/v1/write"
              writeRelabelConfigs:
              - sourceLabels:
                - __tmp_openshift_cluster_id__
                targetLabel: cluster_id
                action: replace
              tlsConfig:
                ca:
                  secret:
                    name: mtls-bundle
                    key: ca.crt
                cert:
                  secret:
                    name: mtls-bundle
                    key: ocp-client.crt
                keySecret:
                  name: mtls-bundle
                  key: ocp-client.key

      1

      Set a retention period. Optionally, you can reduce retention for local metrics because of external collection.

   2. Save the config map as a file called cluster-monitoring-config.yaml.

   3. Create a secret that is named mtls-bundle in the openshift-monitoring namespace that contains HTTPS client certificates for authentication to Prometheus by entering the following command:

      $ oc --namespace openshift-monitoring \
          create secret generic mtls-bundle \
              --from-file=./ca.crt \
              --from-file=ocp-client.crt \
              --from-file=ocp-client.key

   4. Apply the cluster monitoring configuration by running the following command:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Connect to your cluster by using a username and password; do not log in by using a kubeconfig file that was generated by the installation program.

2. To retrieve a token from the OpenShift Container Platform cluster, run the following command on it:

   $ oc whoami -t

3. Make the token available as a secret in the openstack namespace in the RHOSO management cluster by running the following command:

   $ oc -n openstack create secret generic ocp-federated --from-literal=token=<the_token_fetched_previously>

4. To get the Prometheus federation route URL from your OpenShift Container Platform cluster, run the following command:

   $ oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath={'.status.ingress[].host'}

5. Write a manifest for a scrape configuration and save it as a file called cluster-scrape-config.yaml. As an example:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: ScrapeConfig
   metadata:
     labels:
       service: metricStorage
     name: sos1-federated
     namespace: openstack
   spec:
     params:
       'match[]':
       - '{__name__=~"kube_node_info|kube_persistentvolume_info|cluster:master_nodes"}' 1
     metricsPath: '/federate'
     authorization:
       type: Bearer
       credentials:
         name: ocp-federated 2
         key: token
     scheme: HTTPS # or HTTP
     scrapeInterval: 30s 3
     staticConfigs:
     - targets:
       - prometheus-k8s-federate-openshift-monitoring.apps.openshift.example 4

   1

   Add metrics here. In this example, only the metrics kube_node_info, kube_persistentvolume_info, and cluster:master_nodes are requested.

   2

   Insert the previously generated secret name here.

   3

   Limit scraping to fewer than 1000 samples for each request with a maximum frequency of once every 30 seconds.

   4

   Insert the URL you fetched previously here. If the endpoint is HTTPS and uses a custom certificate authority, add a tlsConfig section after it.

6. While connected to the RHOSO management cluster, apply the manifest by running the following command:

   $ oc apply -f cluster-scrape-config.yaml