Cluster Observability Operator

Procedure

1. Add a MonitoringStack resource using the following configuration:

   Example MonitoringStack object

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     labels:
       coo: example
     name: sample-monitoring-stack
     namespace: coo-demo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         app: demo

   Copy to Clipboard Toggle word wrap

2. A Prometheus resource named sample-monitoring-stack is generated in the coo-demo namespace. Retrieve the managed fields of the generated Prometheus resource by running the following command:

   $ oc -n coo-demo get Prometheus.monitoring.rhobs -oyaml --show-managed-fields

   Copy to Clipboard Toggle word wrap

   Example output

   managedFields:
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           f:app.kubernetes.io/managed-by: {}
           f:app.kubernetes.io/name: {}
           f:app.kubernetes.io/part-of: {}
         f:ownerReferences:
           k:{"uid":"81da0d9a-61aa-4df3-affc-71015bcbde5a"}: {}
       f:spec:
         f:additionalScrapeConfigs: {}
         f:affinity:
           f:podAntiAffinity:
             f:requiredDuringSchedulingIgnoredDuringExecution: {}
         f:alerting:
           f:alertmanagers: {}
         f:arbitraryFSAccessThroughSMs: {}
         f:logLevel: {}
         f:podMetadata:
           f:labels:
             f:app.kubernetes.io/component: {}
             f:app.kubernetes.io/part-of: {}
         f:podMonitorSelector: {}
         f:replicas: {}
         f:resources:
           f:limits:
             f:cpu: {}
             f:memory: {}
           f:requests:
             f:cpu: {}
             f:memory: {}
         f:retention: {}
         f:ruleSelector: {}
         f:rules:
           f:alert: {}
         f:securityContext:
           f:fsGroup: {}
           f:runAsNonRoot: {}
           f:runAsUser: {}
         f:serviceAccountName: {}
         f:serviceMonitorSelector: {}
         f:thanos:
           f:baseImage: {}
           f:resources: {}
           f:version: {}
         f:tsdb: {}
     manager: observability-operator
     operation: Apply
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:status:
         .: {}
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
           k:{"type":"Reconciled"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
         f:paused: {}
         f:replicas: {}
         f:shardStatuses:
           .: {}
           k:{"shardID":"0"}:
             .: {}
             f:availableReplicas: {}
             f:replicas: {}
             f:shardID: {}
             f:unavailableReplicas: {}
             f:updatedReplicas: {}
         f:unavailableReplicas: {}
         f:updatedReplicas: {}
     manager: PrometheusOperator
     operation: Update
     subresource: status

   Copy to Clipboard Toggle word wrap

3. Check the metadata.managedFields values, and observe that some fields in metadata and spec are managed by the MonitoringStack resource.

4. Modify a field that is not controlled by the MonitoringStack resource:

   1. Change spec.enforcedSampleLimit, which is a field not set by the MonitoringStack resource. Create the file prom-spec-edited.yaml:

      prom-spec-edited.yaml

      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        enforcedSampleLimit: 1000

      Copy to Clipboard Toggle word wrap

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Copy to Clipboard Toggle word wrap
      Note

      You must use the --server-side flag.

   3. Get the changed Prometheus object and note that there is one more section in managedFields which has spec.enforcedSampleLimit:

      $ oc get prometheus -n coo-demo

      Copy to Clipboard Toggle word wrap

      Example output

      managedFields: 

      1

      
      - apiVersion: monitoring.rhobs/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:app.kubernetes.io/part-of: {}
          f:spec:
            f:enforcedSampleLimit: {} 

      2

      
        manager: kubectl
        operation: Apply

      Copy to Clipboard Toggle word wrap

      1

      managedFields

      2

      spec.enforcedSampleLimit

5. Modify a field that is managed by the MonitoringStack resource:

   1. Change spec.LogLevel, which is a field managed by the MonitoringStack resource, using the following YAML configuration:

      # changing the logLevel from debug to info
      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        logLevel: info 

      1

      Copy to Clipboard Toggle word wrap

      1

      spec.logLevel has been added

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Copy to Clipboard Toggle word wrap

      Example output

      error: Apply failed with 1 conflict: conflict with "observability-operator": .spec.logLevel
      Please review the fields above--they currently have other managers. Here
      are the ways you can resolve this warning:
      * If you intend to manage all of these fields, please re-run the apply
        command with the `--force-conflicts` flag.
      * If you do not intend to manage all of the fields, please edit your
        manifest to remove references to the fields that should keep their
        current managers.
      * You may co-own fields by updating your manifest to match the existing
        value; in this case, you'll become the manager if the other manager(s)
        stop managing the field (remove it from their configuration).
      See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

      Copy to Clipboard Toggle word wrap

   3. Notice that the field spec.logLevel cannot be changed using Server-Side Apply, because it is already managed by observability-operator.

   4. Use the --force-conflicts flag to force the change.

      $ oc apply -f ./prom-spec-edited.yaml --server-side --force-conflicts

      Copy to Clipboard Toggle word wrap

      Example output

      prometheus.monitoring.rhobs/sample-monitoring-stack serverside-applied

      Copy to Clipboard Toggle word wrap

      With --force-conflicts flag, the field can be forced to change, but since the same field is also managed by the MonitoringStack resource, the Observability Operator detects the change, and reverts it back to the value set by the MonitoringStack resource.

      Note

      Some Prometheus fields generated by the MonitoringStack resource are influenced by the fields in the MonitoringStack spec stanza, for example, logLevel. These can be changed by changing the MonitoringStack spec.

   5. To change the logLevel in the Prometheus object, apply the following YAML to change the MonitoringStack resource:

      apiVersion: monitoring.rhobs/v1alpha1
      kind: MonitoringStack
      metadata:
        name: sample-monitoring-stack
        labels:
          coo: example
      spec:
        logLevel: info

      Copy to Clipboard Toggle word wrap

   6. To confirm that the change has taken place, query for the log level by running the following command:

      $ oc -n coo-demo get Prometheus.monitoring.rhobs -o=jsonpath='{.items[0].spec.logLevel}'

      Copy to Clipboard Toggle word wrap

      Example output

      info

      Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Type cluster observability operator in the Filter by keyword box.
3. Click Cluster Observability Operator in the list of results.

4. Read the information about the Operator, and configure the following installation settings:

   • Update channel → stable
   • Version → 1.0.0 or later
   • Installation mode → All namespaces on the cluster (default)
   • Installed Namespace → Operator recommended Namespace: openshift-cluster-observability-operator
   • Select Enable Operator recommended cluster monitoring on this Namespace
   • Update approval → Automatic
5. Optional: You can change the installation settings to suit your requirements. For example, you can select to subscribe to a different update channel, to install an older released version of the Operator, or to require manual approval for updates to new versions of the Operator.
6. Click Install.

Procedure

1. Go to Operators → Installed Operators.
2. Locate the Cluster Observability Operator entry in the list.
3. Click for this entry and select Uninstall Operator.

Procedure

1. Create a YAML file named prometheus-coo-example-app.yaml that contains the following configuration details for a namespace, deployment, and service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP

   Copy to Clipboard Toggle word wrap
2. Save the file.

3. Apply the configuration to the cluster by running the following command:

   $ oc apply -f prometheus-coo-example-app.yaml

   Copy to Clipboard Toggle word wrap

4. Verify that the pod is running by running the following command and observing the output:

   $ oc -n ns1-coo get pod

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-coo-example-app-0927545cb7-anskj   1/1       Running   0          81m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file named example-coo-app-service-monitor.yaml that contains the following ServiceMonitor object configuration details:

   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   Copy to Clipboard Toggle word wrap

   This configuration defines a ServiceMonitor object that the MonitoringStack object will reference to scrape the metrics data exposed by the prometheus-coo-example-app sample service.

2. Apply the configuration to the cluster by running the following command:

   $ oc apply -f example-coo-app-service-monitor.yaml

   Copy to Clipboard Toggle word wrap

3. Verify that the ServiceMonitor resource is created by running the following command and observing the output:

   $ oc -n ns1-coo get servicemonitors.monitoring.rhobs

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                         AGE
   prometheus-coo-example-monitor   81m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for the MonitoringStack object configuration. For this example, name the file example-coo-monitoring-stack.yaml.

2. Add the following MonitoringStack object configuration details:

   Example MonitoringStack object

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         k8s-app: prometheus-coo-example-monitor

   Copy to Clipboard Toggle word wrap

3. Apply the MonitoringStack object by running the following command:

   $ oc apply -f example-coo-monitoring-stack.yaml

   Copy to Clipboard Toggle word wrap

4. Verify that the MonitoringStack object is available by running the following command and inspecting the output:

   $ oc -n ns1-coo get monitoringstack

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                         AGE
   example-coo-monitoring-stack   81m

   Copy to Clipboard Toggle word wrap

5. Run the following comand to retrieve information about the active targets from Prometheus and filter the output to list only targets labeled with app=prometheus-coo-example-app. This verifies which targets are discovered and actively monitored by Prometheus with this specific label.

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'

   Copy to Clipboard Toggle word wrap

   Example output

   {
     "__address__": "10.129.2.25:8080",
     "__meta_kubernetes_endpoint_address_target_kind": "Pod",
     "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_endpoint_port_name": "web",
     "__meta_kubernetes_endpoint_port_protocol": "TCP",
     "__meta_kubernetes_endpoint_ready": "true",
     "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z",
     "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
     "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_endpoints_labelpresent_app": "true",
     "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
     "__meta_kubernetes_namespace": "ns1-coo",
     "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}",
     "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.129.2.25\"\n    ],\n    \"mac\": \"0a:58:0a:81:02:19\",\n    \"default\": true,\n    \"dns\": {}\n}]",
     "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
     "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
     "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
     "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
     "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
     "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
     "__meta_kubernetes_pod_controller_kind": "ReplicaSet",
     "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7",
     "__meta_kubernetes_pod_host_ip": "10.0.128.2",
     "__meta_kubernetes_pod_ip": "10.129.2.25",
     "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7",
     "__meta_kubernetes_pod_labelpresent_app": "true",
     "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
     "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_pod_phase": "Running",
     "__meta_kubernetes_pod_ready": "true",
     "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871",
     "__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_service_labelpresent_app": "true",
     "__meta_kubernetes_service_name": "prometheus-coo-example-app",
     "__metrics_path__": "/metrics",
     "__scheme__": "http",
     "__scrape_interval__": "30s",
     "__scrape_timeout__": "10s",
     "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
   }

   Copy to Clipboard Toggle word wrap

   Note

   The above example uses jq command-line JSON processor to format the output for convenience.

Procedure

1. Create a route to expose the example prometheus-coo-example-app service. From your terminal, run the command:

   $ oc expose svc prometheus-coo-example-app -n ns1-coo

   Copy to Clipboard Toggle word wrap
2. Access the route from your browser, or command line, to generate metrics.

3. Execute a query on the Prometheus pod to return the total HTTP requests metric:

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'

   Copy to Clipboard Toggle word wrap

   Example output (formatted using jq for convenience)

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "200",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "3"
           ]
         },
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "404",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "0"
           ]
         }
       ]
     }
   }

   Copy to Clipboard Toggle word wrap

Procedure

1. Deploy the following namespace object and MonitoringStack YAML file:

   Example MonitoringStack

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
         matchLabels:
           k8s-app: prometheus-coo-example-monitor
     namespaceSelector:
         matchLabels:
           monitoring.rhobs/stack: multi-ns

   Copy to Clipboard Toggle word wrap

2. Deploy a sample application in the namespace ns1-coo, with an alert that is always firing:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app
   ---
   apiVersion: monitoring.rhobs/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1-coo
     labels:
       k8s-app: prometheus-coo-example-monitor
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-coo-example-app"} > 0
         labels:
           severity: warning

   Copy to Clipboard Toggle word wrap

3. Deploy the same example application in another namespace labeled with monitoring.rhobs/stack: multi-ns:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns2-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns2-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that the Prometheus instance adds new targets and that the alert are firing. Use a port-forward command to expose the Prometheus or the Alertmanager user interface that has been deployed by the Monitoringstack instance.

   Prometheus

   $ oc port-forward -n ns1-coo pod/prometheus-example-coo-monitoring-stack-0 9090

   Copy to Clipboard Toggle word wrap

   Alertmanager

   $ oc port-forward -n ns1-coo pod/alertmanager-example-coo-monitoring-stack-0 9093

   Copy to Clipboard Toggle word wrap

2. Verify that the targets are being scraped and that the alerts are firing by browsing to http://localhost:9090/targets or http://localhost:9093/#/alerts.