第 8 章 Troubleshooting and debugging metering

Procedure

1. Check for metering Report resources that fail to complete and show the status of ReportingPeriodUnmetDependencies:

   $ oc get reports

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                  QUERY                          SCHEDULE   RUNNING                            FAILED   LAST REPORT TIME       AGE
   namespace-cpu-utilization-adhoc-10    namespace-cpu-utilization                 Finished                                    2020-10-31T00:00:00Z   2m38s
   namespace-cpu-utilization-adhoc-11    namespace-cpu-utilization                 ReportingPeriodUnmetDependencies                                   2m23s
   namespace-memory-utilization-202010   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   26s
   namespace-memory-utilization-202011   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   14s

   Copy to Clipboard Toggle word wrap

2. Check the ReportDataSource resources where the NEWEST METRIC is less than the report end date:

   $ oc get reportdatasource

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
   ...
   node-allocatable-cpu-cores                   2020-04-23T09:14:00Z   2020-08-31T10:07:00Z   2020-04-23T09:14:00Z   2020-10-15T17:13:00Z   2020-12-09T12:45:10Z   230d
   node-allocatable-memory-bytes                2020-04-23T09:14:00Z   2020-08-30T05:19:00Z   2020-04-23T09:14:00Z   2020-10-14T08:01:00Z   2020-12-09T12:45:12Z   230d
   ...
   pod-usage-memory-bytes                       2020-04-23T09:14:00Z   2020-08-24T20:25:00Z   2020-04-23T09:14:00Z   2020-10-09T23:31:00Z   2020-12-09T12:45:12Z   230d

   Copy to Clipboard Toggle word wrap

3. Check the health of the reporting-operator Pod resource for a high number of pod restarts:

   $ oc get pods -l app=reporting-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                  READY   STATUS    RESTARTS   AGE
   reporting-operator-84f7c9b7b6-fr697   2/2     Running   542        8d 

   1

   Copy to Clipboard Toggle word wrap

   1

   The Reporting Operator pod is restarting at a high rate.

4. Check the reporting-operator Pod resource for an OOMKilled termination:

   $ oc describe pod/reporting-operator-84f7c9b7b6-fr697

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         reporting-operator-84f7c9b7b6-fr697
   Namespace:    openshift-metering
   Priority:     0
   Node:         ip-10-xx-xx-xx.ap-southeast-1.compute.internal/10.xx.xx.xx
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Thu, 03 Dec 2020 20:59:45 +1000
      Last State:     Terminated
        Reason:       OOMKilled 

   1

   
        Exit Code:    137
        Started:      Thu, 03 Dec 2020 20:38:05 +1000
        Finished:     Thu, 03 Dec 2020 20:59:43 +1000

   Copy to Clipboard Toggle word wrap

   1

   The Reporting Operator pod was terminated due to OOM kill.

Procedure

1. Check the current memory limits of the reporting-operator Pod resource:

   $ oc describe pod reporting-operator-67d6f57c56-79mrt

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         reporting-operator-67d6f57c56-79mrt
   Namespace:    openshift-metering
   Priority:     0
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Tue, 08 Dec 2020 14:26:21 +1000
      Ready:          True
      Restart Count:  0
      Limits:
        cpu:     1
        memory:  500Mi 

   1

   
      Requests:
        cpu:     500m
        memory:  250Mi
      Environment:
   ...

   Copy to Clipboard Toggle word wrap

   1

   The current memory limit for the Reporting Operator pod.

2. Edit the MeteringConfig resource to update the memory limit:

   $ oc edit meteringconfig/operator-metering

   Copy to Clipboard Toggle word wrap

   Example MeteringConfig resource

     kind: MeteringConfig
     metadata:
     name: operator-metering
      namespace: openshift-metering
     spec:
      reporting-operator:
        spec:
          resources: 

   1

   
            limits:
              cpu: 1
              memory: 750Mi
            requests:
              cpu: 500m
              memory: 500Mi
     ...

   Copy to Clipboard Toggle word wrap

   1

   Add or increase memory limits within the resources field of the MeteringConfig resource.

   注意

   If there continue to be numerous OOM killed events after memory limits are increased, this might indicate that a different issue is causing the reports to be in a pending state.