Cluster Observability Operator

Procedure

1. Add a MonitoringStack resource using the following configuration:

   Example MonitoringStack object

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     labels:
       coo: example
     name: sample-monitoring-stack
     namespace: coo-demo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         app: demo

2. A Prometheus resource named sample-monitoring-stack is generated in the coo-demo namespace. Retrieve the managed fields of the generated Prometheus resource by running the following command:

   $ oc -n coo-demo get Prometheus.monitoring.rhobs -oyaml --show-managed-fields

   Example output

   managedFields:
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           f:app.kubernetes.io/managed-by: {}
           f:app.kubernetes.io/name: {}
           f:app.kubernetes.io/part-of: {}
         f:ownerReferences:
           k:{"uid":"81da0d9a-61aa-4df3-affc-71015bcbde5a"}: {}
       f:spec:
         f:additionalScrapeConfigs: {}
         f:affinity:
           f:podAntiAffinity:
             f:requiredDuringSchedulingIgnoredDuringExecution: {}
         f:alerting:
           f:alertmanagers: {}
         f:arbitraryFSAccessThroughSMs: {}
         f:logLevel: {}
         f:podMetadata:
           f:labels:
             f:app.kubernetes.io/component: {}
             f:app.kubernetes.io/part-of: {}
         f:podMonitorSelector: {}
         f:replicas: {}
         f:resources:
           f:limits:
             f:cpu: {}
             f:memory: {}
           f:requests:
             f:cpu: {}
             f:memory: {}
         f:retention: {}
         f:ruleSelector: {}
         f:rules:
           f:alert: {}
         f:securityContext:
           f:fsGroup: {}
           f:runAsNonRoot: {}
           f:runAsUser: {}
         f:serviceAccountName: {}
         f:serviceMonitorSelector: {}
         f:thanos:
           f:baseImage: {}
           f:resources: {}
           f:version: {}
         f:tsdb: {}
     manager: observability-operator
     operation: Apply
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:status:
         .: {}
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
           k:{"type":"Reconciled"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
         f:paused: {}
         f:replicas: {}
         f:shardStatuses:
           .: {}
           k:{"shardID":"0"}:
             .: {}
             f:availableReplicas: {}
             f:replicas: {}
             f:shardID: {}
             f:unavailableReplicas: {}
             f:updatedReplicas: {}
         f:unavailableReplicas: {}
         f:updatedReplicas: {}
     manager: PrometheusOperator
     operation: Update
     subresource: status

3. Check the metadata.managedFields values, and observe that some fields in metadata and spec are managed by the MonitoringStack resource.

4. Modify a field that is not controlled by the MonitoringStack resource:

   1. Change spec.enforcedSampleLimit, which is a field not set by the MonitoringStack resource. Create the file prom-spec-edited.yaml:

      prom-spec-edited.yaml

      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        enforcedSampleLimit: 1000

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Note

      You must use the --server-side flag.

   3. Get the changed Prometheus object and note that there is one more section in managedFields which has spec.enforcedSampleLimit:

      $ oc get prometheus -n coo-demo

      Example output

      managedFields: 1
      - apiVersion: monitoring.rhobs/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:app.kubernetes.io/part-of: {}
          f:spec:
            f:enforcedSampleLimit: {} 2
        manager: kubectl
        operation: Apply

      1

      managedFields

      2

      spec.enforcedSampleLimit

5. Modify a field that is managed by the MonitoringStack resource:

   1. Change spec.LogLevel, which is a field managed by the MonitoringStack resource, using the following YAML configuration:

      # changing the logLevel from debug to info
      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        logLevel: info 1

      1

      spec.logLevel has been added

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Example output

      error: Apply failed with 1 conflict: conflict with "observability-operator": .spec.logLevel
      Please review the fields above--they currently have other managers. Here
      are the ways you can resolve this warning:
      * If you intend to manage all of these fields, please re-run the apply
        command with the `--force-conflicts` flag.
      * If you do not intend to manage all of the fields, please edit your
        manifest to remove references to the fields that should keep their
        current managers.
      * You may co-own fields by updating your manifest to match the existing
        value; in this case, you'll become the manager if the other manager(s)
        stop managing the field (remove it from their configuration).
      See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

   3. Notice that the field spec.logLevel cannot be changed using Server-Side Apply, because it is already managed by observability-operator.

   4. Use the --force-conflicts flag to force the change.

      $ oc apply -f ./prom-spec-edited.yaml --server-side --force-conflicts

      Example output

      prometheus.monitoring.rhobs/sample-monitoring-stack serverside-applied

      With --force-conflicts flag, the field can be forced to change, but since the same field is also managed by the MonitoringStack resource, the Observability Operator detects the change, and reverts it back to the value set by the MonitoringStack resource.

      Note

      Some Prometheus fields generated by the MonitoringStack resource are influenced by the fields in the MonitoringStack spec stanza, for example, logLevel. These can be changed by changing the MonitoringStack spec.

   5. To change the logLevel in the Prometheus object, apply the following YAML to change the MonitoringStack resource:

      apiVersion: monitoring.rhobs/v1alpha1
      kind: MonitoringStack
      metadata:
        name: sample-monitoring-stack
        labels:
          coo: example
      spec:
        logLevel: info

   6. To confirm that the change has taken place, query for the log level by running the following command:

      $ oc -n coo-demo get Prometheus.monitoring.rhobs -o=jsonpath='{.items[0].spec.logLevel}'

      Example output

      info

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Type cluster observability operator in the Filter by keyword box.
3. Click Cluster Observability Operator in the list of results.

4. Read the information about the Operator, and configure the following installation settings:

   • Update channel → stable
   • Version → 1.0.0 or later
   • Installation mode → All namespaces on the cluster (default)
   • Installed Namespace → Operator recommended Namespace: openshift-cluster-observability-operator
   • Select Enable Operator recommended cluster monitoring on this Namespace
   • Update approval → Automatic
5. Optional: You can change the installation settings to suit your requirements. For example, you can select to subscribe to a different update channel, to install an older released version of the Operator, or to require manual approval for updates to new versions of the Operator.
6. Click Install.

Procedure

1. Go to Operators → Installed Operators.
2. Locate the Cluster Observability Operator entry in the list.
3. Click for this entry and select Uninstall Operator.

Procedure

1. Create a YAML file named prometheus-coo-example-app.yaml that contains the following configuration details for a namespace, deployment, and service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP

2. Save the file.

3. Apply the configuration to the cluster by running the following command:

   $ oc apply -f prometheus-coo-example-app.yaml

4. Verify that the pod is running by running the following command and observing the output:

   $ oc -n ns1-coo get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-coo-example-app-0927545cb7-anskj   1/1       Running   0          81m

Procedure

1. Create a YAML file named example-coo-app-service-monitor.yaml that contains the following ServiceMonitor object configuration details:

   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   This configuration defines a ServiceMonitor object that the MonitoringStack object will reference to scrape the metrics data exposed by the prometheus-coo-example-app sample service.

2. Apply the configuration to the cluster by running the following command:

   $ oc apply -f example-coo-app-service-monitor.yaml

3. Verify that the ServiceMonitor resource is created by running the following command and observing the output:

   $ oc -n ns1-coo get servicemonitors.monitoring.rhobs

   Example output

   NAME                         AGE
   prometheus-coo-example-monitor   81m

Procedure

1. Create a YAML file for the MonitoringStack object configuration. For this example, name the file example-coo-monitoring-stack.yaml.

2. Add the following MonitoringStack object configuration details:

   Example MonitoringStack object

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         k8s-app: prometheus-coo-example-monitor

3. Apply the MonitoringStack object by running the following command:

   $ oc apply -f example-coo-monitoring-stack.yaml

4. Verify that the MonitoringStack object is available by running the following command and inspecting the output:

   $ oc -n ns1-coo get monitoringstack

   Example output

   NAME                         AGE
   example-coo-monitoring-stack   81m

5. Run the following comand to retrieve information about the active targets from Prometheus and filter the output to list only targets labeled with app=prometheus-coo-example-app. This verifies which targets are discovered and actively monitored by Prometheus with this specific label.

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'

   Example output

   {
     "__address__": "10.129.2.25:8080",
     "__meta_kubernetes_endpoint_address_target_kind": "Pod",
     "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_endpoint_port_name": "web",
     "__meta_kubernetes_endpoint_port_protocol": "TCP",
     "__meta_kubernetes_endpoint_ready": "true",
     "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z",
     "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
     "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_endpoints_labelpresent_app": "true",
     "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
     "__meta_kubernetes_namespace": "ns1-coo",
     "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}",
     "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.129.2.25\"\n    ],\n    \"mac\": \"0a:58:0a:81:02:19\",\n    \"default\": true,\n    \"dns\": {}\n}]",
     "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
     "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
     "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
     "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
     "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
     "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
     "__meta_kubernetes_pod_controller_kind": "ReplicaSet",
     "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7",
     "__meta_kubernetes_pod_host_ip": "10.0.128.2",
     "__meta_kubernetes_pod_ip": "10.129.2.25",
     "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7",
     "__meta_kubernetes_pod_labelpresent_app": "true",
     "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
     "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_pod_phase": "Running",
     "__meta_kubernetes_pod_ready": "true",
     "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871",
     "__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_service_labelpresent_app": "true",
     "__meta_kubernetes_service_name": "prometheus-coo-example-app",
     "__metrics_path__": "/metrics",
     "__scheme__": "http",
     "__scrape_interval__": "30s",
     "__scrape_timeout__": "10s",
     "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
   }

   Note

   The above example uses jq command-line JSON processor to format the output for convenience.

Procedure

1. Create a route to expose the example prometheus-coo-example-app service. From your terminal, run the command:

   $ oc expose svc prometheus-coo-example-app

2. Access the route from your browser, or command line, to generate metrics.

3. Execute a query on the Prometheus pod to return the total HTTP requests metric:

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'

   Example output (formatted using jq for convenience)

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "200",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "3"
           ]
         },
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "404",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "0"
           ]
         }
       ]
     }
   }