Chapter 8. Troubleshooting and debugging metering

Procedure

1. Check for metering

   Report

   resources that fail to complete and show the status of

   ReportingPeriodUnmetDependencies

   :

   $ oc get reports

   Example output

   NAME                                  QUERY                          SCHEDULE   RUNNING                            FAILED   LAST REPORT TIME       AGE
   namespace-cpu-utilization-adhoc-10    namespace-cpu-utilization                 Finished                                    2020-10-31T00:00:00Z   2m38s
   namespace-cpu-utilization-adhoc-11    namespace-cpu-utilization                 ReportingPeriodUnmetDependencies                                   2m23s
   namespace-memory-utilization-202010   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   26s
   namespace-memory-utilization-202011   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   14s

2. Check the

   ReportDataSource

   resources where the

   NEWEST METRIC

   is less than the report end date:

   $ oc get reportdatasource

   Example output

   NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
   ...
   node-allocatable-cpu-cores                   2020-04-23T09:14:00Z   2020-08-31T10:07:00Z   2020-04-23T09:14:00Z   2020-10-15T17:13:00Z   2020-12-09T12:45:10Z   230d
   node-allocatable-memory-bytes                2020-04-23T09:14:00Z   2020-08-30T05:19:00Z   2020-04-23T09:14:00Z   2020-10-14T08:01:00Z   2020-12-09T12:45:12Z   230d
   ...
   pod-usage-memory-bytes                       2020-04-23T09:14:00Z   2020-08-24T20:25:00Z   2020-04-23T09:14:00Z   2020-10-09T23:31:00Z   2020-12-09T12:45:12Z   230d

3. Check the health of the

   reporting-operator

   Pod

   resource for a high number of pod restarts:

   $ oc get pods -l app=reporting-operator

   Example output

   NAME                                  READY   STATUS    RESTARTS   AGE
   reporting-operator-84f7c9b7b6-fr697   2/2     Running   542        8d 

   1

   1

   The Reporting Operator pod is restarting at a high rate.

4. Check the

   reporting-operator

   Pod

   resource for an

   OOMKilled

   termination:

   $ oc describe pod/reporting-operator-84f7c9b7b6-fr697

   Example output

   Name:         reporting-operator-84f7c9b7b6-fr697
   Namespace:    openshift-metering
   Priority:     0
   Node:         ip-10-xx-xx-xx.ap-southeast-1.compute.internal/10.xx.xx.xx
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Thu, 03 Dec 2020 20:59:45 +1000
      Last State:     Terminated
        Reason:       OOMKilled 

   1

   
        Exit Code:    137
        Started:      Thu, 03 Dec 2020 20:38:05 +1000
        Finished:     Thu, 03 Dec 2020 20:59:43 +1000

   1

   The Reporting Operator pod was terminated due to OOM kill.

Procedure

1. Check the current memory limits of the

   reporting-operator

   Pod

   resource:

   $ oc describe pod reporting-operator-67d6f57c56-79mrt

   Example output

   Name:         reporting-operator-67d6f57c56-79mrt
   Namespace:    openshift-metering
   Priority:     0
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Tue, 08 Dec 2020 14:26:21 +1000
      Ready:          True
      Restart Count:  0
      Limits:
        cpu:     1
        memory:  500Mi 

   1

   
      Requests:
        cpu:     500m
        memory:  250Mi
      Environment:
   ...

   1

   The current memory limit for the Reporting Operator pod.

2. Edit the

   MeteringConfig

   resource to update the memory limit:

   $ oc edit meteringconfig/operator-metering

   Example MeteringConfig resource

   kind: MeteringConfig
   metadata:
     name: operator-metering
     namespace: openshift-metering
   spec:
     reporting-operator:
     spec:
       resources: 

   1

   
         limits:
           cpu: 1
           memory: 750Mi
         requests:
           cpu: 500m
           memory: 500Mi
   ...

   1

   Add or increase memory limits within the resources field of the MeteringConfig resource.

   Note

   If there continue to be numerous OOM killed events after memory limits are increased, this might indicate that a different issue is causing the reports to be in a pending state.