5.13. 使用 Prometheus 配置内置监控

流程

1. 在 Operator SDK 项目中，取消注释 config/default/kustomization.yaml 文件中的以下行：

   ../prometheus

   Copy to Clipboard Toggle word wrap

2. 创建自定义控制器类，以便从 Operator 发布其他指标。以下示例将 widgets 和 widgetFailures 收集器声明为全局变量，然后将它们注册到控制器的软件包中的 init() 函数：

   例 5.18. controllers/memcached_controller_test_metrics.go 文件

   package controllers
   
   import (
   	"github.com/prometheus/client_golang/prometheus"
   	"sigs.k8s.io/controller-runtime/pkg/metrics"
   )
   
   
   var (
       widgets = prometheus.NewCounter(
           prometheus.CounterOpts{
               Name: "widgets_total",
               Help: "Number of widgets processed",
           },
       )
       widgetFailures = prometheus.NewCounter(
           prometheus.CounterOpts{
               Name: "widget_failures_total",
               Help: "Number of failed widgets",
           },
       )
   )
   
   func init() {
       // Register custom metrics with the global prometheus registry
       metrics.Registry.MustRegister(widgets, widgetFailures)
   }

   Copy to Clipboard Toggle word wrap

3. 记录这些收集器来自 main 控制器类中协调循环的任何部分，这决定了指标的业务逻辑：

   例 5.19. controllers/memcached_controller.go 文件

   func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
   	...
   	...
   	// Add metrics
   	widgets.Inc()
   	widgetFailures.Inc()
   
   	return ctrl.Result{}, nil
   }

   Copy to Clipboard Toggle word wrap

4. 构建和推送 Operator：

   $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

   Copy to Clipboard Toggle word wrap

5. 部署 Operator：

   $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

   Copy to Clipboard Toggle word wrap

6. 创建角色和角色绑定定义，以允许 OpenShift Container Platform 集群的 Prometheus 实例提取 Operator 的服务监控。

   必须分配角色，以便服务帐户具有提取命名空间指标的权限：

   例 5.20. config/prometheus/role.yaml 角色

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: prometheus-k8s-role
     namespace: <operator_namespace>
   rules:
     - apiGroups:
         - ""
       resources:
         - endpoints
         - pods
         - services
         - nodes
         - secrets
       verbs:
         - get
         - list
         - watch

   Copy to Clipboard Toggle word wrap

   例 5.21. config/prometheus/rolebinding.yaml 角色绑定

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: prometheus-k8s-rolebinding
     namespace: memcached-operator-system
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: prometheus-k8s-role
   subjects:
     - kind: ServiceAccount
       name: prometheus-k8s
       namespace: openshift-monitoring

   Copy to Clipboard Toggle word wrap

7. 为部署的 Operator 应用角色和绑定：

   $ oc apply -f config/prometheus/role.yaml

   Copy to Clipboard Toggle word wrap

   $ oc apply -f config/prometheus/rolebinding.yaml

   Copy to Clipboard Toggle word wrap

8. 为您要提取的命名空间设置标签，为该命名空间启用 OpenShift 集群监控：

   $ oc label namespace <operator_namespace> openshift.io/cluster-monitoring="true"

   Copy to Clipboard Toggle word wrap

流程

1. 生成基于 Ansible 的 Operator。这个示例使用 testmetrics.com 域：

   $ operator-sdk init \
       --plugins=ansible \
       --domain=testmetrics.com

   Copy to Clipboard Toggle word wrap

2. 创建 metrics API。这个示例使用名为 Testmetrics 的 kind ：

   $ operator-sdk create api \
       --group metrics \
       --version v1 \
       --kind Testmetrics \
       --generate-role

   Copy to Clipboard Toggle word wrap

3. 编辑 roles/testmetrics/tasks/main.yml 文件，并使用 osdk_metrics 模块为 Operator 项目创建自定义指标：

   例 5.22. roles/testmetrics/tasks/main.yml 文件示例

   ---
   # tasks file for Memcached
   - name: start k8sstatus
     k8s:
       definition:
         kind: Deployment
         apiVersion: apps/v1
         metadata:
           name: '{{ ansible_operator_meta.name }}-memcached'
           namespace: '{{ ansible_operator_meta.namespace }}'
         spec:
           replicas: "{{size}}"
           selector:
             matchLabels:
               app: memcached
           template:
             metadata:
               labels:
                 app: memcached
             spec:
               containers:
               - name: memcached
                 command:
                 - memcached
                 - -m=64
                 - -o
                 - modern
                 - -v
                 image: "docker.io/memcached:1.4.36-alpine"
                 ports:
                   - containerPort: 11211
   
   - osdk_metric:
       name: my_thing_counter
       description: This metric counts things
       counter: {}
   
   - osdk_metric:
       name: my_counter_metric
       description: Add 3.14 to the counter
       counter:
         increment: yes
   
   - osdk_metric:
       name: my_gauge_metric
       description: Create my gauge and set it to 2.
       gauge:
         set: 2
   
   - osdk_metric:
       name: my_histogram_metric
       description: Observe my histogram
       histogram:
         observe: 2
   
   - osdk_metric:
       name: my_summary_metric
       description: Observe my summary
       summary:
         observe: 2

   Copy to Clipboard Toggle word wrap

验证

1. 在集群上运行 Operator。例如，使用 "run 作为部署" 方法：

   1. 构建 Operator 镜像并将其推送到 registry：

      $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

      Copy to Clipboard Toggle word wrap

   2. 在集群上安装 Operator:

      $ make install

      Copy to Clipboard Toggle word wrap

   3. 部署 Operator：

      $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

      Copy to Clipboard Toggle word wrap

2. 创建 Testmetrics 自定义资源(CR)：

   1. 定义 CR spec：

      例 5.23. config/samples/metrics_v1_testmetrics.yaml 文件示例

      apiVersion: metrics.testmetrics.com/v1
      kind: Testmetrics
      metadata:
        name: testmetrics-sample
      spec:
        size: 1

      Copy to Clipboard Toggle word wrap

   2. 创建对象：

      $ oc create -f config/samples/metrics_v1_testmetrics.yaml

      Copy to Clipboard Toggle word wrap

3. 获取 pod 详情：

   $ oc get pods

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                    READY   STATUS    RESTARTS   AGE
   ansiblemetrics-controller-manager-<id>  2/2     Running   0          149m
   testmetrics-sample-memcached-<id>       1/1     Running   0          147m

   Copy to Clipboard Toggle word wrap

4. 获取端点详情：

   $ oc get ep

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                                ENDPOINTS          AGE
   ansiblemetrics-controller-manager-metrics-service   10.129.2.70:8443   150m

   Copy to Clipboard Toggle word wrap

5. 请求自定义指标令牌：

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`

   Copy to Clipboard Toggle word wrap

6. 检查指标值：

   1. 检查 my_counter_metric 值：

      $ oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza
      tion: Bearer $token" 'https://10.129.2.70:8443/metrics' | grep  my_counter

      Copy to Clipboard Toggle word wrap

      输出示例

      HELP my_counter_metric Add 3.14 to the counter
      TYPE my_counter_metric counter
      my_counter_metric 2

      Copy to Clipboard Toggle word wrap

   2. 检查 my_gauge_metric 值：

      $ oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza
      tion: Bearer $token" 'https://10.129.2.70:8443/metrics' | grep  gauge

      Copy to Clipboard Toggle word wrap

      输出示例

      HELP my_gauge_metric Create my gauge and set it to 2.

      Copy to Clipboard Toggle word wrap

   3. 检查 my_histogram_metric 和 my_summary_metric 值：

      $ oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza
      tion: Bearer $token" 'https://10.129.2.70:8443/metrics' | grep  Observe

      Copy to Clipboard Toggle word wrap

      输出示例

      HELP my_histogram_metric Observe my histogram
      HELP my_summary_metric Observe my summary

      Copy to Clipboard Toggle word wrap