Chapter 14. Scaling Multicloud Object Gateway performance

Prerequisites

• All resources described in this procedure (Roles, RoleBindings, TriggerAuthentication, ScaledObject, and so on) must be created in the openshift-storage namespace.

• Custom Metrics Autoscaler Operator version must be 2.6 or later.

  1. Install the Custom Metrics Autoscaler Operator.

     1. Log in to the OpenShift Web Console.
     2. Click Ecosystem Software Catalog.
     3. Scroll or type Custom Metrics Autoscaler into the Filter by keyword box to find the Custom Metrics Autoscaler Operator.
     4. Click Install.

  2. Create a Custom Metrics Autoscaler controller and verify whether the following pods are running in the openshift-keda namespace:

     oc get pods -n openshift-keda
     NAME                                                  READY   STATUS    RESTARTS   AGE
     custom-metrics-autoscaler-operator-6cb698bb4d-nnpmj   1/1     Running   0          103m
     keda-admission-c6d879546-twprp                        1/1     Running   0          103m
     keda-metrics-apiserver-68788fbd9b-h7l5n               1/1     Running   0          103m
     keda-operator-664dc57859-mb49f                        1/1     Running   0          20m

  3. Configure Custom Metrics Autoscaler with the Thanos Query Service.

     Prometheus scaler that is supported by Custom Metrics Autoscaler collects metrics through the Thanos Query service provided by OpenShift Monitoring. Because Thanos Query requires authentication, a ServiceAccount token must be used for bearer authentication.

     1. Create a Service Account.

        oc create serviceaccount thanos -n openshift-storage

     2. Create a secret to generate the token.

        apiVersion: v1
        kind: Secret
        metadata:
          name: thanos-token-secret
          namespace: openshift-storage
          annotations:
            kubernetes.io/service-account.name: "thanos"
        type: kubernetes.io/service-account-token

        oc apply -f secret.yaml

     3. Add the cluster-monitoring-view cluster role to the created Service Account in the openshift-storage namespace.

        oc adm policy add-cluster-role-to-user cluster-monitoring-view -z thanos

     4. Create a triggerAuthentication CRD for the bearer authentication using the secret name.

        apiVersion: keda.sh/v1alpha1
        kind: TriggerAuthentication
        metadata:
          name: keda-trigger-auth-prometheus
          namespace: openshift-storage
        spec:
          secretTargetRef:
          - parameter: bearerToken
            name: thanos-token-secret
            key: token
          - parameter: ca
            name: thanos-token-secret
            key: ca.crt

  4. Enable RGW autoscaling.

     oc annotate storagecluster ocs-storagecluster ocs.openshift.io/enable-rgw-autoscale="true" -n openshift-storage

     This behavior takes precedence over manual scaling. You can revert this change by performing the following command:

     oc annotate storagecluster ocs-storagecluster ocs.openshift.io/enable-rgw-autoscale- -n openshift-storage

  5. Create ScaledObject for autoscaling RGW.

     Autoscaling using the HPA feature of OpenShift triggers scaling based on custom metrics. The prometheus metrics related to RGW exported by the Ceph manager is used. The following example uses ceph_rgw_put_obj_ops metrics and scale up to five pods.

     apiVersion: keda.sh/v1alpha1
     kind: ScaledObject
     metadata:
       name: rgw-scale-namespace
       namespace: openshift-storage
     spec:
       scaleTargetRef:
         apiVersion: ceph.rook.io/v1
         kind: CephObjectStore
         name: ocs-storagecluster-cephobjectstore
         namespace: openshift-storage
       minReplicaCount: 1
       maxReplicaCount: 5
       triggers:
         - type: prometheus
           metadata:
             serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
             metricName: ceph_rgw_put_obj_ops
             query: |
               sum(rate(ceph_rgw_op_put_obj_ops[2m]))
             threshold: "1"
             authModes: "bearer"
             namespace: openshift-storage
           authenticationRef:
             name: keda-trigger-auth-prometheus

Verification steps

• To verfiy, you can check the resource values of the PV pool pods.