Cluster Observability Operator

流程

1. 使用以下配置添加 MonitoringStack 资源：

   MonitoringStack 对象示例

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     labels:
       coo: example
     name: sample-monitoring-stack
     namespace: coo-demo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         app: demo

   Copy to Clipboard Toggle word wrap

2. coo-demo 命名空间中生成了一个名为 sample-monitoring-stack 的 Prometheus 资源。运行以下命令，检索生成的 Prometheus 资源的受管字段：

   $ oc -n coo-demo get Prometheus.monitoring.rhobs -oyaml --show-managed-fields

   Copy to Clipboard Toggle word wrap

   输出示例

   managedFields:
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           f:app.kubernetes.io/managed-by: {}
           f:app.kubernetes.io/name: {}
           f:app.kubernetes.io/part-of: {}
         f:ownerReferences:
           k:{"uid":"81da0d9a-61aa-4df3-affc-71015bcbde5a"}: {}
       f:spec:
         f:additionalScrapeConfigs: {}
         f:affinity:
           f:podAntiAffinity:
             f:requiredDuringSchedulingIgnoredDuringExecution: {}
         f:alerting:
           f:alertmanagers: {}
         f:arbitraryFSAccessThroughSMs: {}
         f:logLevel: {}
         f:podMetadata:
           f:labels:
             f:app.kubernetes.io/component: {}
             f:app.kubernetes.io/part-of: {}
         f:podMonitorSelector: {}
         f:replicas: {}
         f:resources:
           f:limits:
             f:cpu: {}
             f:memory: {}
           f:requests:
             f:cpu: {}
             f:memory: {}
         f:retention: {}
         f:ruleSelector: {}
         f:rules:
           f:alert: {}
         f:securityContext:
           f:fsGroup: {}
           f:runAsNonRoot: {}
           f:runAsUser: {}
         f:serviceAccountName: {}
         f:serviceMonitorSelector: {}
         f:thanos:
           f:baseImage: {}
           f:resources: {}
           f:version: {}
         f:tsdb: {}
     manager: observability-operator
     operation: Apply
   - apiVersion: monitoring.rhobs/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:status:
         .: {}
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
           k:{"type":"Reconciled"}:
             .: {}
             f:lastTransitionTime: {}
             f:observedGeneration: {}
             f:status: {}
             f:type: {}
         f:paused: {}
         f:replicas: {}
         f:shardStatuses:
           .: {}
           k:{"shardID":"0"}:
             .: {}
             f:availableReplicas: {}
             f:replicas: {}
             f:shardID: {}
             f:unavailableReplicas: {}
             f:updatedReplicas: {}
         f:unavailableReplicas: {}
         f:updatedReplicas: {}
     manager: PrometheusOperator
     operation: Update
     subresource: status

   Copy to Clipboard Toggle word wrap

3. 检查 metadata.managedFields 值，并观察 metadata 和 spec 中的一些字段是否由 MonitoringStack 资源管理。

4. 修改不是由 MonitoringStack 资源控制的字段：

   1. 更改 spec.enforcedSampleLimit，这是 MonitoringStack 资源未设置的字段。创建 prom-spec-edited.yaml 文件：

      prom-spec-edited.yaml

      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        enforcedSampleLimit: 1000

      Copy to Clipboard Toggle word wrap

   2. 运行以下命令来应用 YAML：

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Copy to Clipboard Toggle word wrap
      注意

      您必须使用 --server-side 标志。

   3. 获取更改的 Prometheus 对象，并注意，managedFields 中有多个部分，其 spec.enforcedSampleLimit:

      $ oc get prometheus -n coo-demo

      Copy to Clipboard Toggle word wrap

      输出示例

      managedFields: 

      1

      
      - apiVersion: monitoring.rhobs/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:app.kubernetes.io/part-of: {}
          f:spec:
            f:enforcedSampleLimit: {} 

      2

      
        manager: kubectl
        operation: Apply

      Copy to Clipboard Toggle word wrap

      1

      managedFields

      2

      spec.enforcedSampleLimit

5. 修改由 MonitoringStack 资源管理的字段：

   1. 使用以下 YAML 配置更改 spec.LogLevel，它是由 MonitoringStack 资源管理的字段：

      # changing the logLevel from debug to info
      apiVersion: monitoring.rhobs/v1
      kind: Prometheus
      metadata:
        name: sample-monitoring-stack
        namespace: coo-demo
      spec:
        logLevel: info 

      1

      Copy to Clipboard Toggle word wrap

      1

      添加了 spec.logLevel

   2. 运行以下命令来应用 YAML：

      $ oc apply -f ./prom-spec-edited.yaml --server-side

      Copy to Clipboard Toggle word wrap

      输出示例

      error: Apply failed with 1 conflict: conflict with "observability-operator": .spec.logLevel
      Please review the fields above--they currently have other managers. Here
      are the ways you can resolve this warning:
      * If you intend to manage all of these fields, please re-run the apply
        command with the `--force-conflicts` flag.
      * If you do not intend to manage all of the fields, please edit your
        manifest to remove references to the fields that should keep their
        current managers.
      * You may co-own fields by updating your manifest to match the existing
        value; in this case, you'll become the manager if the other manager(s)
        stop managing the field (remove it from their configuration).
      See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

      Copy to Clipboard Toggle word wrap

   3. 请注意，无法使用 Server-Side Apply 更改字段 spec.logLevel，因为它已经由 observability-operator 管理。

   4. 使用 --force-conflicts 标志来强制更改。

      $ oc apply -f ./prom-spec-edited.yaml --server-side --force-conflicts

      Copy to Clipboard Toggle word wrap

      输出示例

      prometheus.monitoring.rhobs/sample-monitoring-stack serverside-applied

      Copy to Clipboard Toggle word wrap

      with-force-conflicts 标志，字段可以被强制更改，但因为同一字段也由 MonitoringStack 资源管理，所以 Observability Operator 会检测到更改，并将其恢复到 MonitoringStack 资源设置的值。

      注意

      MonitoringStack 资源生成的一些 Prometheus 字段受到 MonitoringStack spec 小节中的字段的影响，如 logLevel。可以通过更改 MonitoringStack spec 来更改它们。

   5. 要更改 Prometheus 对象中的 logLevel，请应用以下 YAML 以更改 MonitoringStack 资源：

      apiVersion: monitoring.rhobs/v1alpha1
      kind: MonitoringStack
      metadata:
        name: sample-monitoring-stack
        labels:
          coo: example
      spec:
        logLevel: info

      Copy to Clipboard Toggle word wrap

   6. 要确认已进行了更改，请运行以下命令来查询日志级别：

      $ oc -n coo-demo get Prometheus.monitoring.rhobs -o=jsonpath='{.items[0].spec.logLevel}'

      Copy to Clipboard Toggle word wrap

      输出示例

      info

      Copy to Clipboard Toggle word wrap

流程

1. 在 OpenShift Container Platform Web 控制台中，点击 Operators → OperatorHub。
2. 在 Filter by keyword 框中键入 cluster observability operator。
3. 在结果列表中点 Cluster Observability Operator。

4. 阅读 Operator 信息并配置以下安装设置：

   • Update channel → stable
   • 版本 → 1.0.0 或更高版本
   • Installation mode → All namespaces on the cluster (default)
   • Installed Namespace → Operator recommended Namespace: openshift-cluster-observability-operator
   • 选择 Enable Operator recommended cluster monitoring on this Namespace
   • Update approval → Automatic
5. 可选： 您可以更改安装设置以满足您的要求。例如，您可以选择订阅不同的更新频道，安装一个旧的 Operator 发行版本，或者需要手动批准对 Operator 的新版本进行更新。
6. 点 Install。

流程

1. 进入 Operators → Installed Operators。
2. 在列表中找到 Cluster Observability Operator 条目。
3. 在这个条目中点 并选择 Uninstall Operator。

流程

1. 创建名为 prometheus-coo-example-app.yaml 的 YAML 文件，其中包含命名空间、部署和服务的以下配置详情：

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP

   Copy to Clipboard Toggle word wrap
2. 保存该文件。

3. 运行以下命令，将配置应用到集群：

   $ oc apply -f prometheus-coo-example-app.yaml

   Copy to Clipboard Toggle word wrap

4. 运行以下命令验证 pod 是否正在运行，并观察输出：

   $ oc -n ns1-coo get pod

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-coo-example-app-0927545cb7-anskj   1/1       Running   0          81m

   Copy to Clipboard Toggle word wrap

流程

1. 创建名为 example-coo-app-service-monitor.yaml 的 YAML 文件，其中包含以下 ServiceMonitor 对象配置详情：

   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   Copy to Clipboard Toggle word wrap

   此配置定义 MonitoringStack 对象将引用的 ServiceMonitor 对象，以提取由 prometheus-coo-example-app 示例服务公开的指标数据。

2. 运行以下命令，将配置应用到集群：

   $ oc apply -f example-coo-app-service-monitor.yaml

   Copy to Clipboard Toggle word wrap

3. 运行以下命令并验证 ServiceMonitor 资源是否已创建并观察输出：

   $ oc -n ns1-coo get servicemonitors.monitoring.rhobs

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                         AGE
   prometheus-coo-example-monitor   81m

   Copy to Clipboard Toggle word wrap

流程

1. 为 MonitoringStack 对象配置创建一个 YAML 文件。在本例中，将文件命名为 example-coo-monitoring-stack.yaml。

2. 添加以下 MonitoringStack 对象配置详情：

   MonitoringStack 对象示例

   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
       matchLabels:
         k8s-app: prometheus-coo-example-monitor

   Copy to Clipboard Toggle word wrap

3. 运行以下命令来应用 MonitoringStack 对象：

   $ oc apply -f example-coo-monitoring-stack.yaml

   Copy to Clipboard Toggle word wrap

4. 运行以下命令并检查输出，验证 MonitoringStack 对象是否可用：

   $ oc -n ns1-coo get monitoringstack

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                         AGE
   example-coo-monitoring-stack   81m

   Copy to Clipboard Toggle word wrap

5. 运行以下 comand，以从 Prometheus 检索有关活动目标的信息，并过滤输出以仅列出标有 app=prometheus-coo-example-app 的目标。这会验证被发现哪些目标，并由 Prometheus 使用这个特定标签主动监控。

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'

   Copy to Clipboard Toggle word wrap

   输出示例

   {
     "__address__": "10.129.2.25:8080",
     "__meta_kubernetes_endpoint_address_target_kind": "Pod",
     "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_endpoint_port_name": "web",
     "__meta_kubernetes_endpoint_port_protocol": "TCP",
     "__meta_kubernetes_endpoint_ready": "true",
     "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z",
     "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
     "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_endpoints_labelpresent_app": "true",
     "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
     "__meta_kubernetes_namespace": "ns1-coo",
     "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}",
     "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.129.2.25\"\n    ],\n    \"mac\": \"0a:58:0a:81:02:19\",\n    \"default\": true,\n    \"dns\": {}\n}]",
     "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
     "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
     "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
     "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
     "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
     "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
     "__meta_kubernetes_pod_controller_kind": "ReplicaSet",
     "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7",
     "__meta_kubernetes_pod_host_ip": "10.0.128.2",
     "__meta_kubernetes_pod_ip": "10.129.2.25",
     "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7",
     "__meta_kubernetes_pod_labelpresent_app": "true",
     "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
     "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
     "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
     "__meta_kubernetes_pod_phase": "Running",
     "__meta_kubernetes_pod_ready": "true",
     "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871",
     "__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
     "__meta_kubernetes_service_labelpresent_app": "true",
     "__meta_kubernetes_service_name": "prometheus-coo-example-app",
     "__metrics_path__": "/metrics",
     "__scheme__": "http",
     "__scrape_interval__": "30s",
     "__scrape_timeout__": "10s",
     "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
   }

   Copy to Clipboard Toggle word wrap

   注意

   上面的示例使用 jq 命令行 JSON 处理器来格式化输出，以方便使用。

流程

1. 创建一个路由，以公开 prometheus-coo-example-app 服务示例。在终端中运行以下命令：

   $ oc expose svc prometheus-coo-example-app -n ns1-coo

   Copy to Clipboard Toggle word wrap
2. 从浏览器或命令行访问路由，以生成指标。

3. 在 Prometheus pod 上执行查询，以返回总 HTTP 请求指标：

   $ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'

   Copy to Clipboard Toggle word wrap

   输出示例（为方便起见使用 jq 进行格式化）

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "200",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "3"
           ]
         },
         {
           "metric": {
             "__name__": "http_requests_total",
             "code": "404",
             "endpoint": "web",
             "instance": "10.129.2.25:8080",
             "job": "prometheus-coo-example-app",
             "method": "get",
             "namespace": "ns1-coo",
             "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
             "service": "prometheus-coo-example-app"
           },
           "value": [
             1730807483.632,
             "0"
           ]
         }
       ]
     }
   }

   Copy to Clipboard Toggle word wrap

流程

1. 部署以下命名空间对象和 MonitoringStack YAML 文件：

   MonitoringStack 示例

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: monitoring.rhobs/v1alpha1
   kind: MonitoringStack
   metadata:
     name: example-coo-monitoring-stack
     namespace: ns1-coo
   spec:
     logLevel: debug
     retention: 1d
     resourceSelector:
         matchLabels:
           k8s-app: prometheus-coo-example-monitor
     namespaceSelector:
         matchLabels:
           monitoring.rhobs/stack: multi-ns

   Copy to Clipboard Toggle word wrap

2. 在命名空间 ns1-coo 中部署示例应用程序，其中包含始终会触发的警报：

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns1-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns1-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app
   ---
   apiVersion: monitoring.rhobs/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1-coo
     labels:
       k8s-app: prometheus-coo-example-monitor
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-coo-example-app"} > 0
         labels:
           severity: warning

   Copy to Clipboard Toggle word wrap

3. 在标有 monitoring.rhobs/stack: multi-ns 的另一个命名空间中部署相同的示例应用程序：

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns2-coo
     labels:
       monitoring.rhobs/stack: multi-ns
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-coo-example-app
     template:
       metadata:
         labels:
           app: prometheus-coo-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-coo-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-coo-example-app
     name: prometheus-coo-example-app
     namespace: ns2-coo
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-coo-example-app
     type: ClusterIP
   ---
   apiVersion: monitoring.rhobs/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-coo-example-monitor
     name: prometheus-coo-example-monitor
     namespace: ns2-coo
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-coo-example-app

   Copy to Clipboard Toggle word wrap

验证

1. 验证 Prometheus 实例是否已添加新目标，以及警报是否已触发。使用 port-forward 命令公开 Prometheus 或 Monitoringstack 实例部署的 Alertmanager 用户界面。

   Prometheus

   $ oc port-forward -n ns1-coo pod/prometheus-example-coo-monitoring-stack-0 9090

   Copy to Clipboard Toggle word wrap

   Alertmanager

   $ oc port-forward -n ns1-coo pod/alertmanager-example-coo-monitoring-stack-0 9093

   Copy to Clipboard Toggle word wrap

2. 通过浏览 http://localhost:9090/targets 或 http://localhost:9093/#/alerts 来验证目标是否已被 scrape。