Chapter 2. Cluster Observability Operator overview

Procedure

1. Add a MonitoringStack resource using the following configuration:

   Example MonitoringStack object

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     labels:
       coo: example
     name: sample-monitoring-stack
     namespace: coo-demo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         app: demo

2. A Prometheus resource named sample-monitoring-stack is generated in the coo-demo namespace. Retrieve the managed fields of the generated Prometheus resource by running the following command:

   $ oc -n coo-demo get Prometheus.monitoring.rhobs -oyaml --show-managed-fields

   Example output

   managedFields:
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           f:app.kubernetes.io/managed-by: {}
           f:app.kubernetes.io/name: {}
           f:app.kubernetes.io/part-of: {}
         f:ownerReferences:
           k:{"uid":"81da0d9a-61aa-4df3-affc-71015bcbde5a"}: {}
       f:spec:
         f:additionalScrapeConfigs: {}
         f:affinity:
           f:podAntiAffinity:
             f:requiredDuringSchedulingIgnoredDuringExecution: {}
         f:alerting:
           f:alertmanagers: {}
         f:arbitraryFSAccessThroughSMs: {}
         f:logLevel: {}
         f:podMetadata:
           f:labels:
             f:app.kubernetes.io/component: {}
             f:app.kubernetes.io/part-of: {}
         f:podMonitorSelector: {}
         f:replicas: {}
         f:resources:
           f:limits:
             f:cpu: {}
             f:memory: {}
           f:requests:
             f:cpu: {}
             f:memory: {}
         f:retention: {}
         f:ruleSelector: {}
         f:rules:
           f:alert: {}
         f:securityContext:
           f:fsGroup: {}
           f:runAsNonRoot: {}
           f:runAsUser: {}
         f:serviceAccountName: {}
         f:serviceMonitorSelector: {}
         f:thanos:
           f:baseImage: {}
           f:resources: {}
           f:version: {}
         f:tsdb: {}
     manager: observability-operator
     operation: Apply
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:status:
         .: {}
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
           k:{"type":"Reconciled"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
         f:paused: {}
         f:replicas: {}
         f:shardStatuses:
           .: {}
           k:{"shardID":"0"}:
             .: {}
             f:availableReplicas: {}
             f:replicas: {}
             f:shardID: {}
             f:unavailableReplicas: {}
             f:updatedReplicas: {}
         f:unavailableReplicas: {}
         f:updatedReplicas: {}
     manager: PrometheusOperator
     operation: Update
     subresource: status

3. Check the metadata.managedFields values, and observe that some fields in metadata and spec are managed by the MonitoringStack resource.

4. Modify a field that is not controlled by the MonitoringStack resource:

   1. Change spec.enforcedSampleLimit, which is a field not set by the MonitoringStack resource. Create the file prom-spec-edited.yaml:

      prom-spec-edited.yaml

      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        enforcedSampleLimit: 1000

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Note

      You must use the --server-side flag.

   3. Get the changed Prometheus object and note that there is one more section in managedFields which has spec.enforcedSampleLimit:

      $ oc get prometheus -n coo-demo

      Example output

      managedFields: 1
      - apiVersion: monitoring.rhobs/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:app.kubernetes.io/part-of: {}
          f:spec:
            f:enforcedSampleLimit: {} 2
        manager: kubectl
        operation: Apply

      1

      managedFields

      2

      spec.enforcedSampleLimit

5. Modify a field that is managed by the MonitoringStack resource:

   1. Change spec.LogLevel, which is a field managed by the MonitoringStack resource, using the following YAML configuration:

      # changing the logLevel from debug to info
      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        logLevel: info 1

      1

      spec.logLevel has been added

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Example output

      error: Apply failed with 1 conflict: conflict with "observability-operator": .spec.logLevel
      Please review the fields above--they currently have other managers. Here
      are the ways you can resolve this warning:
      * If you intend to manage all of these fields, please re-run the apply
        command with the `--force-conflicts` flag.
      * If you do not intend to manage all of the fields, please edit your
        manifest to remove references to the fields that should keep their
        current managers.
      * You may co-own fields by updating your manifest to match the existing
        value; in this case, you'll become the manager if the other manager(s)
        stop managing the field (remove it from their configuration).
      See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

   3. Notice that the field spec.logLevel cannot be changed using Server-Side Apply, because it is already managed by observability-operator.

   4. Use the --force-conflicts flag to force the change.

      $ oc apply -f ./prom-spec-edited.yaml --server-side --force-conflicts

      Example output

      prometheus.monitoring.rhobs/sample-monitoring-stack serverside-applied

      With --force-conflicts flag, the field can be forced to change, but since the same field is also managed by the MonitoringStack resource, the Observability Operator detects the change, and reverts it back to the value set by the MonitoringStack resource.

      Note

      Some Prometheus fields generated by the MonitoringStack resource are influenced by the fields in the MonitoringStack spec stanza, for example, logLevel. These can be changed by changing the MonitoringStack spec.

   5. To change the logLevel in the Prometheus object, apply the following YAML to change the MonitoringStack resource:

      apiVersion: monitoring.rhobs/v1alpha1
      kind: MonitoringStack
      metadata:
        name: sample-monitoring-stack
        labels:
          coo: example
      spec:
        logLevel: info

   6. To confirm that the change has taken place, query for the log level by running the following command:

      $ oc -n coo-demo get Prometheus.monitoring.rhobs -o=jsonpath='{.items[0].spec.logLevel}'

      Example output

      info