第 8 章 metering 故障排除与调试

流程

1. 检查未完成的、显示 ReportingPeriodUnmetDependencies 状态的 metering Report 资源：

   $ oc get reports

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                  QUERY                          SCHEDULE   RUNNING                            FAILED   LAST REPORT TIME       AGE
   namespace-cpu-utilization-adhoc-10    namespace-cpu-utilization                 Finished                                    2020-10-31T00:00:00Z   2m38s
   namespace-cpu-utilization-adhoc-11    namespace-cpu-utilization                 ReportingPeriodUnmetDependencies                                   2m23s
   namespace-memory-utilization-202010   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   26s
   namespace-memory-utilization-202011   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   14s

   Copy to Clipboard Toggle word wrap

2. 检查 NEWEST METRIC 小于报告结束日期的 ReportDataSource 资源：

   $ oc get reportdatasource

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
   ...
   node-allocatable-cpu-cores                   2020-04-23T09:14:00Z   2020-08-31T10:07:00Z   2020-04-23T09:14:00Z   2020-10-15T17:13:00Z   2020-12-09T12:45:10Z   230d
   node-allocatable-memory-bytes                2020-04-23T09:14:00Z   2020-08-30T05:19:00Z   2020-04-23T09:14:00Z   2020-10-14T08:01:00Z   2020-12-09T12:45:12Z   230d
   ...
   pod-usage-memory-bytes                       2020-04-23T09:14:00Z   2020-08-24T20:25:00Z   2020-04-23T09:14:00Z   2020-10-09T23:31:00Z   2020-12-09T12:45:12Z   230d

   Copy to Clipboard Toggle word wrap

3. 检查 reporting-operator Pod 资源的健康状态，以检查大量 pod 重启：

   $ oc get pods -l app=reporting-operator

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                  READY   STATUS    RESTARTS   AGE
   reporting-operator-84f7c9b7b6-fr697   2/2     Running   542        8d 

   1

   Copy to Clipboard Toggle word wrap

   1

   Reporting Operator pod 以更高的率重启。

4. 检查 reporting-operator Pod 资源是否有 OOMKilled 终止：

   $ oc describe pod/reporting-operator-84f7c9b7b6-fr697

   Copy to Clipboard Toggle word wrap

   输出示例

   Name:         reporting-operator-84f7c9b7b6-fr697
   Namespace:    openshift-metering
   Priority:     0
   Node:         ip-10-xx-xx-xx.ap-southeast-1.compute.internal/10.xx.xx.xx
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Thu, 03 Dec 2020 20:59:45 +1000
      Last State:     Terminated
        Reason:       OOMKilled 

   1

   
        Exit Code:    137
        Started:      Thu, 03 Dec 2020 20:38:05 +1000
        Finished:     Thu, 03 Dec 2020 20:59:43 +1000

   Copy to Clipboard Toggle word wrap

   1

   Reporting Operator pod 因 OOM 终止而终止。

流程

1. 检查 reporting-operator Pod 资源的当前内存限值：

   $ oc describe pod reporting-operator-67d6f57c56-79mrt

   Copy to Clipboard Toggle word wrap

   输出示例

   Name:         reporting-operator-67d6f57c56-79mrt
   Namespace:    openshift-metering
   Priority:     0
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Tue, 08 Dec 2020 14:26:21 +1000
      Ready:          True
      Restart Count:  0
      Limits:
        cpu:     1
        memory:  500Mi 

   1

   
      Requests:
        cpu:     500m
        memory:  250Mi
      Environment:
   ...

   Copy to Clipboard Toggle word wrap

   1

   Reporting Operator pod 的当前内存限值。

2. 编辑 MeteringConfig 资源以更新内存限制：

   $ oc edit meteringconfig/operator-metering

   Copy to Clipboard Toggle word wrap

   MeteringConfig 资源示例

   kind: MeteringConfig
   metadata:
     name: operator-metering
     namespace: openshift-metering
   spec:
     reporting-operator:
     spec:
       resources: 

   1

   
         limits:
           cpu: 1
           memory: 750Mi
         requests:
           cpu: 500m
           memory: 500Mi
   ...

   Copy to Clipboard Toggle word wrap

   1

   在 MeteringConfig 资源的 resources 字段中添加或增加内存限值。

   注意

   如果在增加内存限制后仍然有大量 OOM 终止事件，这可能代表报告处于待处理状态。