Metering

Procedure

1. Create a namespace object YAML file for the Metering Operator with the oc create -f <file-name>.yaml command. You must use the CLI to create the namespace. For example, metering-namespace.yaml:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-metering 

   1

   
     annotations:
       openshift.io/node-selector: "" 

   2

   
     labels:
       openshift.io/cluster-monitoring: "true"

   Copy to Clipboard Toggle word wrap

   1

   It is strongly recommended to deploy metering in the openshift-metering namespace.

   2

   Include this annotation before configuring specific node selectors for the operand pods.

2. In the OpenShift Container Platform web console, click Operators → OperatorHub. Filter for metering to find the Metering Operator.
3. Click the Metering card, review the package description, and then click Install.
4. Select an Update Channel, Installation Mode, and Approval Strategy.
5. Click Install.

6. Verify that the Metering Operator is installed by switching to the Operators → Installed Operators page. The Metering Operator has a Status of Succeeded when the installation is complete.

   Note

   It might take several minutes for the Metering Operator to appear.

7. Click Metering on the Installed Operators page for Operator Details. From the Details page you can create different resources related to metering.

Procedure

1. Create a Namespace object YAML file for the Metering Operator. You must use the CLI to create the namespace. For example, metering-namespace.yaml:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-metering 

   1

   
     annotations:
       openshift.io/node-selector: "" 

   2

   
     labels:
       openshift.io/cluster-monitoring: "true"

   Copy to Clipboard Toggle word wrap

   1

   It is strongly recommended to deploy metering in the openshift-metering namespace.

   2

   Include this annotation before configuring specific node selectors for the operand pods.

2. Create the Namespace object:

   $ oc create -f <file-name>.yaml

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc create -f openshift-metering.yaml

   Copy to Clipboard Toggle word wrap

3. Create the OperatorGroup object YAML file. For example, metering-og:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-metering 

   1

   
     namespace: openshift-metering 

   2

   
   spec:
     targetNamespaces:
     - openshift-metering

   Copy to Clipboard Toggle word wrap

   1

   The name is arbitrary.

   2

   Specify the openshift-metering namespace.

4. Create a Subscription object YAML file to subscribe a namespace to the Metering Operator. This object targets the most recently released version in the redhat-operators catalog source. For example, metering-sub.yaml:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metering-ocp 

   1

   
     namespace: openshift-metering 

   2

   
   spec:
     channel: "4.7" 

   3

   
     source: "redhat-operators" 

   4

   
     sourceNamespace: "openshift-marketplace"
     name: "metering-ocp"
     installPlanApproval: "Automatic" 

   5

   Copy to Clipboard Toggle word wrap

   1

   The name is arbitrary.

   2

   You must specify the openshift-metering namespace.

   3

   Specify 4.7 as the channel.

   4

   Specify the redhat-operators catalog source, which contains the metering-ocp package manifests. If your OpenShift Container Platform is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator LifeCycle Manager (OLM).

   5

   Specify "Automatic" install plan approval.

Procedure

1. Edit the spec.storage section in the s3-storage.yaml file:

   Example s3-storage.yaml file

   apiVersion: metering.openshift.io/v1
   kind: MeteringConfig
   metadata:
     name: "operator-metering"
   spec:
     storage:
       type: "hive"
       hive:
         type: "s3"
         s3:
           bucket: "bucketname/path/" 

   1

   
           region: "us-west-1" 

   2

   
           secretName: "my-aws-secret" 

   3

   
           # Set to false if you want to provide an existing bucket, instead of
           # having metering create the bucket on your behalf.
           createBucket: true 

   4

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the bucket where you would like to store your data. Optional: Specify the path within the bucket.

   2

   Specify the region of your bucket.

   3

   The name of a secret in the metering namespace containing the AWS credentials in the data.aws-access-key-id and data.aws-secret-access-key fields. See the example Secret object below for more details.

   4

   Set this field to false if you want to provide an existing S3 bucket, or if you do not want to provide IAM credentials that have CreateBucket permissions.

2. Use the following Secret object as a template:

   Example AWS Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-aws-secret
   data:
     aws-access-key-id: "dGVzdAo="
     aws-secret-access-key: "c2VjcmV0Cg=="

   Copy to Clipboard Toggle word wrap

   Note

   The values of the aws-access-key-id and aws-secret-access-key must be base64 encoded.

3. Create the secret:

   $ oc create secret -n openshift-metering generic my-aws-secret \
     --from-literal=aws-access-key-id=my-access-key \
     --from-literal=aws-secret-access-key=my-secret-key

   Copy to Clipboard Toggle word wrap
   Note

   This command automatically base64 encodes your aws-access-key-id and aws-secret-access-key values.

Procedure

1. Edit the spec.storage section in the s3-compatible-storage.yaml file:

   Example s3-compatible-storage.yaml file

   apiVersion: metering.openshift.io/v1
   kind: MeteringConfig
   metadata:
     name: "operator-metering"
   spec:
     storage:
       type: "hive"
       hive:
         type: "s3Compatible"
         s3Compatible:
           bucket: "bucketname" 

   1

   
           endpoint: "http://example:port-number" 

   2

   
           secretName: "my-aws-secret" 

   3

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of your S3-compatible bucket.

   2

   Specify the endpoint for your storage.

   3

   The name of a secret in the metering namespace containing the AWS credentials in the data.aws-access-key-id and data.aws-secret-access-key fields. See the example Secret object below for more details.

2. Use the following Secret object as a template:

   Example S3-compatible Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-aws-secret
   data:
     aws-access-key-id: "dGVzdAo="
     aws-secret-access-key: "c2VjcmV0Cg=="

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the spec.storage section in the azure-blob-storage.yaml file:

   Example azure-blob-storage.yaml file

   apiVersion: metering.openshift.io/v1
   kind: MeteringConfig
   metadata:
     name: "operator-metering"
   spec:
     storage:
       type: "hive"
       hive:
         type: "azure"
         azure:
           container: "bucket1" 

   1

   
           secretName: "my-azure-secret" 

   2

   
           rootDirectory: "/testDir" 

   3

   Copy to Clipboard Toggle word wrap

   1

   Specify the container name.

   2

   Specify a secret in the metering namespace. See the example Secret object below for more details.

   3

   Optional: Specify the directory where you would like to store your data.

2. Use the following Secret object as a template:

   Example Azure Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-azure-secret
   data:
     azure-storage-account-name: "dGVzdAo="
     azure-secret-access-key: "c2VjcmV0Cg=="

   Copy to Clipboard Toggle word wrap

3. Create the secret:

   $ oc create secret -n openshift-metering generic my-azure-secret \
     --from-literal=azure-storage-account-name=my-storage-account-name \
     --from-literal=azure-secret-access-key=my-secret-key

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the spec.storage section in the gcs-storage.yaml file:

   Example gcs-storage.yaml file

   apiVersion: metering.openshift.io/v1
   kind: MeteringConfig
   metadata:
     name: "operator-metering"
   spec:
     storage:
       type: "hive"
       hive:
         type: "gcs"
         gcs:
           bucket: "metering-gcs/test1" 

   1

   
           secretName: "my-gcs-secret" 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the bucket. You can optionally specify the directory within the bucket where you would like to store your data.

   2

   Specify a secret in the metering namespace. See the example Secret object below for more details.

2. Use the following Secret object as a template:

   Example Google Cloud Storage Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-gcs-secret
   data:
     gcs-service-account.json: "c2VjcmV0Cg=="

   Copy to Clipboard Toggle word wrap

3. Create the secret:

   $ oc create secret -n openshift-metering generic my-gcs-secret \
     --from-file gcs-service-account.json=/path/to/my/service-account-key.json

   Copy to Clipboard Toggle word wrap

Procedure

1. Modify the shared-storage.yaml file to use a ReadWriteMany persistent volume for storage:

   apiVersion: metering.openshift.io/v1
   kind: MeteringConfig
   metadata:
     name: "operator-metering"
   spec:
     storage:
       type: "hive"
       hive:
         type: "sharedPVC"
         sharedPVC:
           claimName: "metering-nfs" 

   1

   
           # Uncomment the lines below to provision a new PVC using the specified storageClass. 

   2

   
           # createPVC: true
           # storageClass: "my-nfs-storage-class"
           # size: 5Gi

   Copy to Clipboard Toggle word wrap

   Select one of the configuration options below:

   1

   Set storage.hive.sharedPVC.claimName to the name of an existing ReadWriteMany persistent volume claim (PVC). This configuration is necessary if you do not have dynamic volume provisioning or want to have more control over how the persistent volume is created.

   2

   Set storage.hive.sharedPVC.createPVC to true and set the storage.hive.sharedPVC.storageClass to the name of a storage class with ReadWriteMany access mode. This configuration uses dynamic volume provisioning to create a volume automatically.

2. Create the following resource objects that are required to deploy an NFS server for metering. Use the oc create -f <file-name>.yaml command to create the object YAML files.

   1. Configure a PersistentVolume resource object:

      Example nfs_persistentvolume.yaml file

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: nfs
        labels:
          role: nfs-server
      spec:
        capacity:
          storage: 5Gi
        accessModes:
        - ReadWriteMany
        storageClassName: nfs-server 

      1

      
        nfs:
          path: "/"
          server: REPLACEME
        persistentVolumeReclaimPolicy: Delete

      Copy to Clipboard Toggle word wrap

      1

      Must exactly match the [kind: StorageClass].metadata.name field value.

   2. Configure a Pod resource object with the nfs-server role:

      Example nfs_server.yaml file

      apiVersion: v1
      kind: Pod
      metadata:
        name: nfs-server
        labels:
          role: nfs-server
      spec:
        containers:
          - name: nfs-server
            image: <image_name> 

      1

      
            imagePullPolicy: IfNotPresent
            ports:
              - name: nfs
                containerPort: 2049
            securityContext:
              privileged: true
            volumeMounts:
            - mountPath: "/mnt/data"
              name: local
        volumes:
          - name: local
            emptyDir: {}

      Copy to Clipboard Toggle word wrap

      1

      Install your NFS server image.

   3. Configure a Service resource object with the nfs-server role:

      Example nfs_service.yaml file

      apiVersion: v1
      kind: Service
      metadata:
        name: nfs-service
        labels:
          role: nfs-server
      spec:
        ports:
        - name: 2049-tcp
          port: 2049
          protocol: TCP
          targetPort: 2049
        selector:
          role: nfs-server
        sessionAffinity: None
        type: ClusterIP

      Copy to Clipboard Toggle word wrap

   4. Configure a StorageClass resource object:

      Example nfs_storageclass.yaml file

      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: nfs-server 

      1

      
      provisioner: example.com/nfs
      parameters:
        archiveOnDelete: "false"
      reclaimPolicy: Delete
      volumeBindingMode: Immediate

      Copy to Clipboard Toggle word wrap

      1

      Must exactly match the [kind: PersistentVolume].spec.storageClassName field value.

Procedure

1. Create a secret using the OpenShift CLI (oc) or by using a YAML file:

   • Create a secret by using the following command:

     $ oc --namespace openshift-metering create secret generic <YOUR_SECRETNAME> --from-literal=username=<YOUR_DATABASE_USERNAME> --from-literal=password=<YOUR_DATABASE_PASSWORD>

     Copy to Clipboard Toggle word wrap

   • Create a secret by using a YAML file. For example:

     apiVersion: v1
     kind: Secret
     metadata:
       name: <YOUR_SECRETNAME> 

     1

     
     data:
       username: <BASE64_ENCODED_DATABASE_USERNAME> 

     2

     
       password: <BASE64_ENCODED_DATABASE_PASSWORD> 

     3

     Copy to Clipboard Toggle word wrap

     1

     The name of the secret.

     2

     Base64 encoded database user name.

     3

     Base64 encoded database password.

2. Create a configuration file to use a MySQL or PostgreSQL database for Hive:

   • To use a MySQL database for Hive, use the example configuration file below. Metering supports configuring the internal Hive metastore to use the MySQL server versions 5.6, 5.7, and 8.0.

     spec:
       hive:
         spec:
           metastore:
             storage:
               create: false
           config:
             db:
               url: "jdbc:mysql://mysql.example.com:3306/hive_metastore" 

     1

     
               driver: "com.mysql.cj.jdbc.Driver"
               secretName: "REPLACEME" 

     2

     Copy to Clipboard Toggle word wrap
     Note

     When configuring Metering to work with older MySQL server versions, such as 5.6 or 5.7, you might need to add the enabledTLSProtocols JDBC URL parameter when configuring the internal Hive metastore.

     1

     To use the TLS v1.2 cipher suite, set url to "jdbc:mysql://<hostname>:<port>/<schema>?enabledTLSProtocols=TLSv1.2".

     2

     The name of the secret containing the base64-encrypted user name and password database credentials.

     You can pass additional JDBC parameters using the spec.hive.config.url. For more details, see the MySQL Connector/J 8.0 documentation.

   • To use a PostgreSQL database for Hive, use the example configuration file below:

     spec:
       hive:
         spec:
           metastore:
             storage:
               create: false
           config:
             db:
               url: "jdbc:postgresql://postgresql.example.com:5432/hive_metastore"
               driver: "org.postgresql.Driver"
               username: "REPLACEME"
               password: "REPLACEME"

     Copy to Clipboard Toggle word wrap

     You can pass additional JDBC parameters using the spec.hive.config.url. For more details, see the PostgreSQL JDBC driver documentation.

Procedure

1. Check for metering Report resources that fail to complete and show the status of ReportingPeriodUnmetDependencies:

   $ oc get reports

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                  QUERY                          SCHEDULE   RUNNING                            FAILED   LAST REPORT TIME       AGE
   namespace-cpu-utilization-adhoc-10    namespace-cpu-utilization                 Finished                                    2020-10-31T00:00:00Z   2m38s
   namespace-cpu-utilization-adhoc-11    namespace-cpu-utilization                 ReportingPeriodUnmetDependencies                                   2m23s
   namespace-memory-utilization-202010   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   26s
   namespace-memory-utilization-202011   namespace-memory-utilization              ReportingPeriodUnmetDependencies                                   14s

   Copy to Clipboard Toggle word wrap

2. Check the ReportDataSource resources where the NEWEST METRIC is less than the report end date:

   $ oc get reportdatasource

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
   ...
   node-allocatable-cpu-cores                   2020-04-23T09:14:00Z   2020-08-31T10:07:00Z   2020-04-23T09:14:00Z   2020-10-15T17:13:00Z   2020-12-09T12:45:10Z   230d
   node-allocatable-memory-bytes                2020-04-23T09:14:00Z   2020-08-30T05:19:00Z   2020-04-23T09:14:00Z   2020-10-14T08:01:00Z   2020-12-09T12:45:12Z   230d
   ...
   pod-usage-memory-bytes                       2020-04-23T09:14:00Z   2020-08-24T20:25:00Z   2020-04-23T09:14:00Z   2020-10-09T23:31:00Z   2020-12-09T12:45:12Z   230d

   Copy to Clipboard Toggle word wrap

3. Check the health of the reporting-operator Pod resource for a high number of pod restarts:

   $ oc get pods -l app=reporting-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                  READY   STATUS    RESTARTS   AGE
   reporting-operator-84f7c9b7b6-fr697   2/2     Running   542        8d 

   1

   Copy to Clipboard Toggle word wrap

   1

   The Reporting Operator pod is restarting at a high rate.

4. Check the reporting-operator Pod resource for an OOMKilled termination:

   $ oc describe pod/reporting-operator-84f7c9b7b6-fr697

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         reporting-operator-84f7c9b7b6-fr697
   Namespace:    openshift-metering
   Priority:     0
   Node:         ip-10-xx-xx-xx.ap-southeast-1.compute.internal/10.xx.xx.xx
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Thu, 03 Dec 2020 20:59:45 +1000
      Last State:     Terminated
        Reason:       OOMKilled 

   1

   
        Exit Code:    137
        Started:      Thu, 03 Dec 2020 20:38:05 +1000
        Finished:     Thu, 03 Dec 2020 20:59:43 +1000

   Copy to Clipboard Toggle word wrap

   1

   The Reporting Operator pod was terminated due to OOM kill.

Procedure

1. Check the current memory limits of the reporting-operator Pod resource:

   $ oc describe pod reporting-operator-67d6f57c56-79mrt

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         reporting-operator-67d6f57c56-79mrt
   Namespace:    openshift-metering
   Priority:     0
   ...
      Ports:          8080/TCP, 6060/TCP, 8082/TCP
      Host Ports:     0/TCP, 0/TCP, 0/TCP
      State:          Running
        Started:      Tue, 08 Dec 2020 14:26:21 +1000
      Ready:          True
      Restart Count:  0
      Limits:
        cpu:     1
        memory:  500Mi 

   1

   
      Requests:
        cpu:     500m
        memory:  250Mi
      Environment:
   ...

   Copy to Clipboard Toggle word wrap

   1

   The current memory limit for the Reporting Operator pod.

2. Edit the MeteringConfig resource to update the memory limit:

   $ oc edit meteringconfig/operator-metering

   Copy to Clipboard Toggle word wrap

   Example MeteringConfig resource

   kind: MeteringConfig
   metadata:
     name: operator-metering
     namespace: openshift-metering
   spec:
     reporting-operator:
     spec:
       resources: 

   1

   
         limits:
           cpu: 1
           memory: 750Mi
         requests:
           cpu: 500m
           memory: 500Mi
   ...

   Copy to Clipboard Toggle word wrap

   1

   Add or increase memory limits within the resources field of the MeteringConfig resource.

   Note

   If there continue to be numerous OOM killed events after memory limits are increased, this might indicate that a different issue is causing the reports to be in a pending state.