About Red Hat OpenShift Cluster Observability Operator

Procedure

1. Add a

   MonitoringStack

   resource using the following configuration:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     labels:
       coo: example
     name: sample-monitoring-stack
     namespace: coo-demo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         app: demo

2. A Prometheus resource named

   sample-monitoring-stack

   is generated in the

   coo-demo

   namespace. Retrieve the managed fields of the generated Prometheus resource by running the following command:

   $ oc -n coo-demo get Prometheus.monitoring.rhobs -oyaml --show-managed-fields

   Example output:

   managedFields:
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           f:app.kubernetes.io/managed-by: {}
           f:app.kubernetes.io/name: {}
           f:app.kubernetes.io/part-of: {}
         f:ownerReferences:
           k:{"uid":"81da0d9a-61aa-4df3-affc-71015bcbde5a"}: {}
       f:spec:
         f:additionalScrapeConfigs: {}
         f:affinity:
           f:podAntiAffinity:
             f:requiredDuringSchedulingIgnoredDuringExecution: {}
         f:alerting:
           f:alertmanagers: {}
         f:arbitraryFSAccessThroughSMs: {}
         f:logLevel: {}
         f:podMetadata:
           f:labels:
             f:app.kubernetes.io/component: {}
             f:app.kubernetes.io/part-of: {}
         f:podMonitorSelector: {}
         f:replicas: {}
         f:resources:
           f:limits:
             f:cpu: {}
             f:memory: {}
           f:requests:
             f:cpu: {}
             f:memory: {}
         f:retention: {}
         f:ruleSelector: {}
         f:rules:
           f:alert: {}
         f:securityContext:
           f:fsGroup: {}
           f:runAsNonRoot: {}
           f:runAsUser: {}
         f:serviceAccountName: {}
         f:serviceMonitorSelector: {}
         f:thanos:
           f:baseImage: {}
           f:resources: {}
           f:version: {}
         f:tsdb: {}
     manager: observability-operator
     operation: Apply
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:status:
         .: {}
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
           k:{"type":"Reconciled"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
         f:paused: {}
         f:replicas: {}
         f:shardStatuses:
           .: {}
           k:{"shardID":"0"}:
             .: {}
             f:availableReplicas: {}
             f:replicas: {}
             f:shardID: {}
             f:unavailableReplicas: {}
             f:updatedReplicas: {}
         f:unavailableReplicas: {}
         f:updatedReplicas: {}
     manager: PrometheusOperator
     operation: Update
     subresource: status

3. Check the

   metadata.managedFields

   values, and observe that some fields in

   metadata

   and

   spec

   are managed by the

   MonitoringStack

   resource.

4. Modify a field that is not controlled by the

   MonitoringStack

   resource:

   1. Change

      spec.enforcedSampleLimit

      , which is a field not set by the

      MonitoringStack

      resource. Create the file

      prom-spec-edited.yaml

      :

      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        enforcedSampleLimit: 1000

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Note

      You must use the

      --server-side

      flag.

   3. Get the changed Prometheus object and note that there is one more section in

      managedFields

      which has

      spec.enforcedSampleLimit

      :

      $ oc get prometheus -n coo-demo

      Example output:

      managedFields:
      - apiVersion: monitoring.rhobs/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:app.kubernetes.io/part-of: {}
          f:spec:
            f:enforcedSampleLimit: {}
        manager: kubectl
        operation: Apply

5. Modify a field that is managed by the

   MonitoringStack

   resource:

   1. Change

      spec.LogLevel

      , which is a field managed by the

      MonitoringStack

      resource, using the following YAML configuration:

      # changing the logLevel from debug to info
      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        logLevel: info

      spec.logLevel

      Added and set to info

   2. Apply the YAML by running the following command:

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Example output:

      error: Apply failed with 1 conflict: conflict with "observability-operator": .spec.logLevel
      Please review the fields above--they currently have other managers. Here
      are the ways you can resolve this warning:
      * If you intend to manage all of these fields, please re-run the apply
        command with the `--force-conflicts` flag.
      * If you do not intend to manage all of the fields, please edit your
        manifest to remove references to the fields that should keep their
        current managers.
      * You may co-own fields by updating your manifest to match the existing
        value; in this case, you'll become the manager if the other manager(s)
        stop managing the field (remove it from their configuration).
      See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

   3. Notice that the field

      spec.logLevel

      cannot be changed using Server-Side Apply, because it is already managed by

      observability-operator

      .

   4. Use the

      --force-conflicts

      flag to force the change.

      $ oc apply -f ./prom-spec-edited.yaml --server-side --force-conflicts

      Example output:

      prometheus.monitoring.rhobs/sample-monitoring-stack serverside-applied

      With

      --force-conflicts

      flag, the field can be forced to change, but since the same field is also managed by the

      MonitoringStack

      resource, the Observability Operator detects the change, and reverts it back to the value set by the

      MonitoringStack

      resource.
      Note

      Some Prometheus fields generated by the

      MonitoringStack

      resource are influenced by the fields in the

      MonitoringStack

      spec

      stanza, for example,

      logLevel

      . These can be changed by changing the

      MonitoringStack

      spec

      .

   5. To change the

      logLevel

      in the Prometheus object, apply the following YAML to change the

      MonitoringStack

      resource:

      apiVersion: monitoring.rhobs/v1alpha1
      kind: MonitoringStack
      metadata:
        name: sample-monitoring-stack
        labels:
          coo: example
      spec:
        logLevel: info

   6. To confirm that the change has taken place, query for the log level by running the following command:

      $ oc -n coo-demo get Prometheus.monitoring.rhobs -o=jsonpath='{.items[0].spec.logLevel}'

      Example output:

      info