Installing Red Hat OpenShift Cluster Observability Operator

Procedure

1. In the OpenShift Container Platform web console:

   1. Click Operators → OperatorHub, if your version of OpenShift Container Platform is 4.19 or earlier.
   2. Click Ecosystem → Software Catalog, if your version of OpenShift Container Platform is 4.20+.
2. Type cluster observability operator in the Filter by keyword box.
3. Click Cluster Observability Operator in the list of results.

4. Read the information about the Operator, and configure the following installation settings:

   • Update channel → stable
   • Version → 1.0.0 or later
   • Installation mode → All namespaces on the cluster (default)
   • Installed Namespace → Operator recommended Namespace: openshift-cluster-observability-operator
   • Select Enable Operator recommended cluster monitoring on this Namespace
   • Update approval → Automatic
5. Optional: You can change the installation settings to suit your requirements. For example, you can select to subscribe to a different update channel, to install an older released version of the Operator, or to require manual approval for updates to new versions of the Operator.
6. Click Install.

Procedure

1. In the OpenShift Container Platform web console:

   1. Go to Operators → Installed Operators, if your version of OpenShift Container Platform is 4.19 or earlier.
   2. Go to Ecosystem → Installed Operators, if your version of OpenShift Container Platform is 4.20+.
2. Locate the Cluster Observability Operator entry in the list.
3. Click for this entry and select Uninstall Operator.

Procedure

1. Install distributed tracing by using the ObservabilityInstaller resource.

   apiVersion: observability.openshift.io/v1alpha1
   kind: ObservabilityInstaller
   metadata:
     name: tracing
     namespace: observability
   spec:
     capabilities:
       tracing:
         enabled: true
         operators:
           install: true
         storage:
           objectStorage:
   ...

   namespace: observability

   Namespace where the ObservabilityInstaller CR is created. The namespace should contain the secrets referenced in the object storage configuration.

   enabled: true

   Enables the tracing capability. This installs the OpenTelemetry Collector, Tempo and all necessary configuration.

   install: true

   Installs the operators required by the tracing capability. The default is true.

   objectStorage:

   Required object storage configuration for the tracing capability. This is used to store traces in an object storage service.

2. Configure S3 object storage

         storage:
           objectStorage:
             s3:
               bucket: <bucket_name>
               endpoint: <s3_endpoint>
               accessKeyID: <access_key_id>
               accessKeySecret:
                 name: s3-access-secret
                 key: access_key_secret
               region: us-east-1

   <bucket_name>

   Name of the bucket to store data in.

   <s3_endpoint>

   Endpoint of the S3-compatible object storage service.

   <access_key_id>

   Access key ID for the S3-compatible object storage service.

   <access_key_secret>

   Secret containing the access key secret for the S3-compatible object storage service.

   <region>

   AWS region for the S3 bucket.

   Use the following command to create the secret containing the access key:

   $ oc create secret generic s3-access-secret \
   --from-literal=access_key_secret="secret"

Procedure

1. Create a YAML file named prometheus-coo-example-app.yaml that contains the following configuration details for a namespace, deployment, and service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP

2. Save the file.

3. Apply the configuration to the cluster by running the following command:

   $ oc apply -f prometheus-coo-example-app.yaml

4. Verify that the pod is running by running the following command and observing the output:

   $ oc -n ns1-coo get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-coo-example-app-0927545cb7-anskj   1/1       Running   0          81m

Procedure

1. Create a YAML file named example-coo-app-service-monitor.yaml that contains the following ServiceMonitor object configuration details:

   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   This configuration defines a ServiceMonitor object that the MonitoringStack object will reference to scrape the metrics data exposed by the prometheus-coo-example-app sample service.

2. Apply the configuration to the cluster by running the following command:

   $ oc apply -f example-coo-app-service-monitor.yaml

3. Verify that the ServiceMonitor resource is created by running the following command and observing the output:

   $ oc -n ns1-coo get servicemonitors.monitoring.rhobs

   Example output

   NAME                         AGE
   prometheus-coo-example-monitor   81m

Procedure

1. Create a YAML file for the MonitoringStack object configuration. For this example, name the file example-coo-monitoring-stack.yaml.

2. Add the following MonitoringStack object configuration details:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         k8s-app: prometheus-coo-example-monitor

3. Apply the MonitoringStack object by running the following command:

   $ oc apply -f example-coo-monitoring-stack.yaml

4. Verify that the MonitoringStack object is available by running the following command and inspecting the output:

   $ oc -n ns1-coo get monitoringstack

   Example output

   NAME                         AGE
   example-coo-monitoring-stack   81m

5. Run the following comand to retrieve information about the active targets from Prometheus and filter the output to list only targets labeled with app=prometheus-coo-example-app. This verifies which targets are discovered and actively monitored by Prometheus with this specific label.

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'

   Example output

   {
     "__address__": "10.129.2.25:8080",
     "__meta_kubernetes_endpoint_address_target_kind": "Pod",
     "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_endpoint_port_name": "web",
     "__meta_kubernetes_endpoint_port_protocol": "TCP",
     "__meta_kubernetes_endpoint_ready": "true",
     "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z",
     "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
     "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_endpoints_labelpresent_app": "true",
     "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
     "__meta_kubernetes_namespace": "ns1-coo",
     "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}",
     "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.129.2.25\"\n    ],\n    \"mac\": \"0a:58:0a:81:02:19\",\n    \"default\": true,\n    \"dns\": {}\n}]",
     "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
     "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
     "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
     "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
     "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
     "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
     "__meta_kubernetes_pod_controller_kind": "ReplicaSet",
     "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7",
     "__meta_kubernetes_pod_host_ip": "10.0.128.2",
     "__meta_kubernetes_pod_ip": "10.129.2.25",
     "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7",
     "__meta_kubernetes_pod_labelpresent_app": "true",
     "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
     "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_pod_phase": "Running",
     "__meta_kubernetes_pod_ready": "true",
     "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871",
     "__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_service_labelpresent_app": "true",
     "__meta_kubernetes_service_name": "prometheus-coo-example-app",
     "__metrics_path__": "/metrics",
     "__scheme__": "http",
     "__scrape_interval__": "30s",
     "__scrape_timeout__": "10s",
     "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
   }

   Note

   The above example uses jq command-line JSON processor to format the output for convenience.

Procedure

1. Create a route to expose the example prometheus-coo-example-app service. From your terminal, run the command:

   $ oc expose svc prometheus-coo-example-app -n ns1-coo

2. Access the route from your browser, or command line, to generate metrics.

3. Execute a query on the Prometheus pod to return the total HTTP requests metric:

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'

   Example output (formatted using jq for convenience)

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "200",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "3"
           ]
         },
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "404",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "0"
           ]
         }
       ]
     }
   }

Procedure

1. Deploy the following namespace object and MonitoringStack YAML file:

   Example MonitoringStack

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
         matchLabels:
           k8s-app: prometheus-coo-example-monitor
     namespaceSelector:
         matchLabels:
           monitoring.rhobs/stack: multi-ns

2. Deploy a sample application in the namespace ns1-coo, with an alert that is always firing:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app
   ---
   apiVersion: monitoring.rhobs/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1-coo
     labels:
       k8s-app: prometheus-coo-example-monitor
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-coo-example-app"} > 0
         labels:
           severity: warning

3. Deploy the same example application in another namespace labeled with monitoring.rhobs/stack: multi-ns:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns2-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns2-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

Verification

1. Verify that the Prometheus instance adds new targets and that the alert are firing. Use a port-forward command to expose the Prometheus or the Alertmanager user interface that has been deployed by the Monitoringstack instance.

   Prometheus

   $ oc port-forward -n ns1-coo pod/prometheus-example-coo-monitoring-stack-0 9090

   Alertmanager

   $ oc port-forward -n ns1-coo pod/alertmanager-example-coo-monitoring-stack-0 9093

2. Verify that the targets are being scraped and that the alerts are firing by browsing to http://localhost:9090/targets or http://localhost:9093/#/alerts.