Chapter 3. Automatically scaling pods with the Custom Metrics Autoscaler Operator

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose Custom Metrics Autoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the All namespaces on the cluster (default) option is selected for Installation Mode. This installs the Operator in all namespaces.
4. Ensure that the openshift-keda namespace is selected for Installed Namespace. OpenShift Container Platform creates the namespace, if not present in your cluster.
5. Click Install.

6. Verify the installation by listing the Custom Metrics Autoscaler Operator components:

   1. Navigate to Workloads Pods.
   2. Select the

      openshift-keda

      project from the drop-down menu and verify that the

      custom-metrics-autoscaler-operator-*

      pod is running.
   3. Navigate to Workloads Deployments to verify that the

      custom-metrics-autoscaler-operator

      deployment is running.

7. Optional: Verify the installation in the OpenShift CLI using the following commands:

   $ oc get all -n openshift-keda

   The output appears similar to the following:

   Example output

   NAME                                                      READY   STATUS    RESTARTS   AGE
   pod/custom-metrics-autoscaler-operator-5fd8d9ffd8-xt4xp   1/1     Running   0          18m
   
   NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/custom-metrics-autoscaler-operator   1/1     1            1           18m
   
   NAME                                                            DESIRED   CURRENT   READY   AGE
   replicaset.apps/custom-metrics-autoscaler-operator-5fd8d9ffd8   1         1         1       18m

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. Click Custom Metrics Autoscaler.
3. On the Operator Details page, click the KedaController tab.

4. On the KedaController tab, click Create KedaController and edit the file.

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: openshift-keda
   spec:
     watchNamespace: '' 

   1

   
     operator:
       logLevel: info 

   2

   
       logEncoder: console 

   3

   
       caConfigMaps: 

   4

   
       - thanos-cert
       - kafka-cert
       volumeMounts: 

   5

   
       - mountPath: /<path_to_directory>
         name: <name>
       volumes: 

   6

   
       - name: <volume_name>
         emptyDir:
           medium: Memory
     metricsServer:
       logLevel: '0' 

   7

   
       auditConfig: 

   8

   
         logFormat: "json"
         logOutputVolumeClaim: "persistentVolumeClaimName"
         policy:
           rules:
           - level: Metadata
           omitStages: ["RequestReceived"]
           omitManagedFields: false
         lifetime:
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"
     serviceAccount: {}

   1

   Specifies a single namespace in which the Custom Metrics Autoscaler Operator scales applications. Leave it blank or leave it empty to scale applications in all namespaces. This field should have a namespace or be empty. The default value is empty.

   2

   Specifies the level of verbosity for the Custom Metrics Autoscaler Operator log messages. The allowed values are debug, info, error. The default is info.

   3

   Specifies the logging format for the Custom Metrics Autoscaler Operator log messages. The allowed values are console or json. The default is console.

   4

   Optional: Specifies one or more config maps with CA certificates, which the Custom Metrics Autoscaler Operator can use to connect securely to TLS-enabled metrics sources.

   5

   Optional: Add the container mount path.

   6

   Optional: Add a volumes block to list each projected volume source.

   7

   Specifies the logging level for the Custom Metrics Autoscaler Metrics Server. The allowed values are 0 for info and 4 for debug. The default is 0.

   8

   Activates audit logging for the Custom Metrics Autoscaler Operator and specifies the audit policy to use, as described in the "Configuring audit logging" section.

5. Click Save to save the changes.

Procedure

1. Create the

   TriggerAuthentication

   or

   ClusterTriggerAuthentication

   object.

   1. Create a YAML file that defines the object:

      Example trigger authentication with a bound service account token

      kind: TriggerAuthentication
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: prom-triggerauthentication
        namespace: my-namespace 

      1

      
        spec:
        boundServiceAccountToken: 

      2

      
          - parameter: token
            serviceAccountName: thanos 

      3

      1

      Specifies the namespace of the object you want to scale.

      2

      Specifies that this trigger authentication uses a bound service account token for authorization when connecting to the metrics endpoint.

      3

      Specifies the name of the service account to use.

   2. Create the

      TriggerAuthentication

      object:

      $ oc create -f <filename>.yaml

2. Create or edit a

   ScaledObject

   YAML file that uses the trigger authentication:

   1. Create a YAML file that defines the object by running the following command:

      Example scaled object with a trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-triggerauthentication 

      1

      
            kind: TriggerAuthentication 

      2

      1

      Specify the name of your trigger authentication object.

      2

      Specify TriggerAuthentication. TriggerAuthentication is the default.

      Example scaled object with a cluster trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-cluster-triggerauthentication 

      1

      
            kind: ClusterTriggerAuthentication 

      2

      1

      Specify the name of your trigger authentication object.

      2

      Specify ClusterTriggerAuthentication.

   2. Create the scaled object by running the following command:

      $ oc apply -f <filename>

Procedure

1. Create a YAML file similar to the following. Only the name

   <2>

   , object name

   <4>

   , and object kind

   <5>

   are required:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "0" 

   1

   
     name: scaledobject 

   2

   
     namespace: my-namespace
   spec:
     scaleTargetRef:
       apiVersion: apps/v1 

   3

   
       name: example-deployment 

   4

   
       kind: Deployment 

   5

   
       envSourceContainerName: .spec.template.spec.containers[0] 

   6

   
     cooldownPeriod:  200 

   7

   
     maxReplicaCount: 100 

   8

   
     minReplicaCount: 0 

   9

   
     metricsServer: 

   10

   
       auditConfig:
         logFormat: "json"
         logOutputVolumeClaim: "persistentVolumeClaimName"
         policy:
           rules:
           - level: Metadata
           omitStages: "RequestReceived"
           omitManagedFields: false
         lifetime:
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"
     fallback: 

   11

   
       failureThreshold: 3
       replicas: 6
       behavior: static 

   12

   
     pollingInterval: 30 

   13

   
     advanced:
       restoreToOriginalReplicaCount: false 

   14

   
       horizontalPodAutoscalerConfig:
         name: keda-hpa-scale-down 

   15

   
         behavior: 

   16

   
           scaleDown:
             stabilizationWindowSeconds: 300
             policies:
             - type: Percent
               value: 100
               periodSeconds: 15
     triggers:
     - type: prometheus 

   17

   
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: basic
       authenticationRef: 

   18

   
         name: prom-triggerauthentication
         kind: TriggerAuthentication

   1

   Optional: Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling, as described in the "Pausing the custom metrics autoscaler for a workload" section.

   2

   Specifies a name for this custom metrics autoscaler.

   3

   Optional: Specifies the API version of the target resource. The default is apps/v1.

   4

   Specifies the name of the object that you want to scale.

   5

   Specifies the kind as Deployment, StatefulSet or CustomResource.

   6

   Optional: Specifies the name of the container in the target resource, from which the custom metrics autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].

   7

   Optional. Specifies the period in seconds to wait after the last trigger is reported before scaling the deployment back to 0 if the minReplicaCount is set to 0. The default is 300.

   8

   Optional: Specifies the maximum number of replicas when scaling up. The default is 100.

   9

   Optional: Specifies the minimum number of replicas when scaling down.

   10

   Optional: Specifies the parameters for audit logs. as described in the "Configuring audit logging" section.

   11

   Optional: Specifies the number of replicas to fall back to if a scaler fails to get metrics from the source for the number of times defined by the failureThreshold parameter. For more information on fallback behavior, see the KEDA documentation.

   12

   Optional: Specifies the replica count to be used if a fallback occurs. Enter one of the following options or omit the parameter:

   • Enter

     static

     to use the number of replicas specified by the

     fallback.replicas

     parameter. This is the default.
   • Enter

     currentReplicas

     to maintain the current number of replicas.
   • Enter

     currentReplicasIfHigher

     to maintain the current number of replicas, if that number is higher than the

     fallback.replicas

     parameter. If the current number of replicas is lower than the

     fallback.replicas

     parameter, use the

     fallback.replicas

     value.
   • Enter

     currentReplicasIfLower

     to maintain the current number of replicas, if that number is lower than the

     fallback.replicas

     parameter. If the current number of replicas is higher than the

     fallback.replicas

     parameter, use the

     fallback.replicas

     value.

   13

   Optional: Specifies the interval in seconds to check each trigger on. The default is 30.

   14

   Optional: Specifies whether to scale back the target resource to the original replica count after the scaled object is deleted. The default is false, which keeps the replica count as it is when the scaled object is deleted.

   15

   Optional: Specifies a name for the horizontal pod autoscaler. The default is keda-hpa-{scaled-object-name}.

   16

   Optional: Specifies a scaling policy to use to control the rate to scale pods up or down, as described in the "Scaling policies" section.

   17

   Specifies the trigger to use as the basis for scaling, as described in the "Understanding the custom metrics autoscaler triggers" section. This example uses OpenShift Container Platform monitoring.

   18

   Optional: Specifies a trigger authentication or a cluster trigger authentication. For more information, see Understanding the custom metrics autoscaler trigger authentication in the Additional resources section.

   • Enter

     TriggerAuthentication

     to use a trigger authentication. This is the default.
   • Enter

     ClusterTriggerAuthentication

     to use a cluster trigger authentication.

2. Create the custom metrics autoscaler by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. Create a YAML file similar to the following:

   kind: ScaledJob
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: scaledjob
     namespace: my-namespace
   spec:
     failedJobsHistoryLimit: 5
     jobTargetRef:
       activeDeadlineSeconds: 600 

   1

   
       backoffLimit: 6 

   2

   
       parallelism: 1 

   3

   
       completions: 1 

   4

   
       template:  

   5

   
         metadata:
           name: pi
         spec:
           containers:
           - name: pi
             image: perl
             command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
     maxReplicaCount: 100 

   6

   
     pollingInterval: 30 

   7

   
     successfulJobsHistoryLimit: 5 

   8

   
     failedJobsHistoryLimit: 5 

   9

   
     envSourceContainerName: 

   10

   
     rolloutStrategy: gradual 

   11

   
     scalingStrategy: 

   12

   
       strategy: "custom"
       customScalingQueueLengthDeduction: 1
       customScalingRunningJobPercentage: "0.5"
       pendingPodConditions:
         - "Ready"
         - "PodScheduled"
         - "AnyOtherCustomPodCondition"
       multipleScalersCalculation : "max"
     triggers:
     - type: prometheus 

   13

   
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "bearer"
       authenticationRef: 

   14

   
         name: prom-cluster-triggerauthentication

   1

   Specifies the maximum duration the job can run.

   2

   Specifies the number of retries for a job. The default is 6.

   3

   Optional: Specifies how many pod replicas a job should run in parallel; defaults to 1.

   • For non-parallel jobs, leave unset. When unset, the default is

     1

     .

   4

   Optional: Specifies how many successful pod completions are needed to mark a job completed.

   • For non-parallel jobs, leave unset. When unset, the default is

     1

     .
   • For parallel jobs with a fixed completion count, specify the number of completions.
   • For parallel jobs with a work queue, leave unset. When unset the default is the value of the

     parallelism

     parameter.

   5

   Specifies the template for the pod the controller creates.

   6

   Optional: Specifies the maximum number of replicas when scaling up. The default is 100.

   7

   Optional: Specifies the interval in seconds to check each trigger on. The default is 30.

   8

   Optional: Specifies the number of successful finished jobs should be kept. The default is 100.

   9

   Optional: Specifies how many failed jobs should be kept. The default is 100.

   10

   Optional: Specifies the name of the container in the target resource, from which the custom autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].

   11

   Optional: Specifies whether existing jobs are terminated whenever a scaled job is being updated:

   • default

     : The autoscaler terminates an existing job if its associated scaled job is updated. The autoscaler recreates the job with the latest specs.

   • gradual

     : The autoscaler does not terminate an existing job if its associated scaled job is updated. The autoscaler creates new jobs with the latest specs.

   12

   Optional: Specifies a scaling strategy: default, custom, or accurate. The default is default. For more information, see the link in the "Additional resources" section that follows.

   13

   Specifies the trigger to use as the basis for scaling, as described in the "Understanding the custom metrics autoscaler triggers" section.

   14

   Optional: Specifies a trigger authentication or a cluster trigger authentication. For more information, see Understanding the custom metrics autoscaler trigger authentication in the Additional resources section.

   • Enter

     TriggerAuthentication

     to use a trigger authentication. This is the default.
   • Enter

     ClusterTriggerAuthentication

     to use a cluster trigger authentication.

2. Create the custom metrics autoscaler by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. Use the following command to edit the

   ScaledObject

   CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Add the

   autoscaling.keda.sh/paused-replicas

   annotation with any value:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 

   1

   
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling.

Procedure

1. Use the following command to edit the

   ScaledObject

   CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Remove the

   autoscaling.keda.sh/paused-replicas

   annotation.

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 

   1

   
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Remove this annotation to restart a paused custom metrics autoscaler.

Procedure

1. Edit the

   KedaController

   custom resource to add the

   auditConfig

   stanza:

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: openshift-keda
   spec:
   # ...
     metricsServer:
   # ...
       auditConfig:
         logFormat: "json" 

   1

   
         logOutputVolumeClaim: "pvc-audit-log" 

   2

   
         policy:
           rules: 

   3

   
           - level: Metadata
           omitStages: "RequestReceived" 

   4

   
           omitManagedFields: false 

   5

   
         lifetime: 

   6

   
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"

   1

   Specifies the output format of the audit log, either legacy or json.

   2

   Specifies an existing persistent volume claim for storing the log data. All requests coming to the API server are logged to this persistent volume claim. If you leave this field empty, the log data is sent to stdout.

   3

   Specifies which events should be recorded and what data they should include:

   • None

     : Do not log events.

   • Metadata

     : Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. This is the default.

   • Request

     : Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.

   • RequestResponse

     : Log event metadata, request text, and response text. This option does not apply for non-resource requests.

   4

   Specifies stages for which no event is created.

   5

   Specifies whether to omit the managed fields of the request and response bodies from being written to the API audit log, either true to omit the fields or false to include the fields.

   6

   Specifies the size and lifespan of the audit logs.

   • maxAge

     : The maximum number of days to retain audit log files, based on the timestamp encoded in their filename.

   • maxBackup

     : The maximum number of audit log files to retain. Set to

     0

     to retain all audit log files.

   • maxSize

     : The maximum size in megabytes of an audit log file before it gets rotated.

Verification

1. View the audit log file directly:

   1. Obtain the name of the

      keda-metrics-apiserver-*

      pod:

      oc get pod -n openshift-keda

      Example output

      NAME                                                  READY   STATUS    RESTARTS   AGE
      custom-metrics-autoscaler-operator-5cb44cd75d-9v4lv   1/1     Running   0          8m20s
      keda-metrics-apiserver-65c7cc44fd-rrl4r               1/1     Running   0          2m55s
      keda-operator-776cbb6768-zpj5b                        1/1     Running   0          2m55s

   2. View the log data by using a command similar to the following:

      $ oc logs keda-metrics-apiserver-<hash>|grep -i metadata 

      1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      $ oc logs keda-metrics-apiserver-65c7cc44fd-rrl4r|grep -i metadata

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4c81d41b-3dab-4675-90ce-20b87ce24013","stage":"ResponseComplete","requestURI":"/healthz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.131.0.1"],"userAgent":"kube-probe/1.27","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-02-16T13:00:03.554567Z","stageTimestamp":"2023-02-16T13:00:03.555032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
       ...

2. Alternatively, you can view a specific log:

   1. Use a command similar to the following to log into the

      keda-metrics-apiserver-*

      pod:

      $ oc rsh pod/keda-metrics-apiserver-<hash> -n openshift-keda

      For example:

      $ oc rsh pod/keda-metrics-apiserver-65c7cc44fd-rrl4r -n openshift-keda

   2. Change to the

      /var/audit-policy/

      directory:

      sh-4.4$ cd /var/audit-policy/

   3. List the available logs:

      sh-4.4$ ls

      Example output

      log-2023.02.17-14:50  policy.yaml

   4. View the log, as needed:

      sh-4.4$ cat <log_name>/<pvc_name>|grep -i <log_level> 

      1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      sh-4.4$ cat log-2023.02.17-14:50/pvc-audit-log|grep -i Request

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"63e7f68c-04ec-4f4d-8749-bf1656572a41","stage":"ResponseComplete","requestURI":"/openapi/v2","verb":"get","user":{"username":"system:aggregator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"responseStatus":{"metadata":{},"code":304},"requestReceivedTimestamp":"2023-02-17T13:12:55.035478Z","stageTimestamp":"2023-02-17T13:12:55.038346Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
       ...

Procedure

1. Navigate to the directory where you want to store the

   must-gather

   data.
   Note

   If your cluster is using a restricted network, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters on restricted networks, you must import the default

   must-gather

   image as an image stream by running the following command.

   $ oc import-image is/must-gather -n openshift

2. Perform one of the following:

   • To get only the Custom Metrics Autoscaler Operator

     must-gather

     data, use the following command:

     $ oc adm must-gather --image="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
     -n openshift-marketplace \
     -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     The custom image for the

     must-gather

     command is pulled directly from the Operator package manifests, so that it works on any cluster where the Custom Metric Autoscaler Operator is available.

   • To gather the default

     must-gather

     data in addition to the Custom Metric Autoscaler Operator information:

     1. Use the following command to obtain the Custom Metrics Autoscaler Operator image and set it as an environment variable:

        $ IMAGE="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
          -n openshift-marketplace \
          -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     2. Use the

        oc adm must-gather

        with the Custom Metrics Autoscaler Operator image:

        $ oc adm must-gather --image-stream=openshift/must-gather --image=${IMAGE}

   Example 3.1. Example must-gather output for the Custom Metric Autoscaler:

   └── openshift-keda
       ├── apps
       │   ├── daemonsets.yaml
       │   ├── deployments.yaml
       │   ├── replicasets.yaml
       │   └── statefulsets.yaml
       ├── apps.openshift.io
       │   └── deploymentconfigs.yaml
       ├── autoscaling
       │   └── horizontalpodautoscalers.yaml
       ├── batch
       │   ├── cronjobs.yaml
       │   └── jobs.yaml
       ├── build.openshift.io
       │   ├── buildconfigs.yaml
       │   └── builds.yaml
       ├── core
       │   ├── configmaps.yaml
       │   ├── endpoints.yaml
       │   ├── events.yaml
       │   ├── persistentvolumeclaims.yaml
       │   ├── pods.yaml
       │   ├── replicationcontrollers.yaml
       │   ├── secrets.yaml
       │   └── services.yaml
       ├── discovery.k8s.io
       │   └── endpointslices.yaml
       ├── image.openshift.io
       │   └── imagestreams.yaml
       ├── k8s.ovn.org
       │   ├── egressfirewalls.yaml
       │   └── egressqoses.yaml
       ├── keda.sh
       │   ├── kedacontrollers
       │   │   └── keda.yaml
       │   ├── scaledobjects
       │   │   └── example-scaledobject.yaml
       │   └── triggerauthentications
       │       └── example-triggerauthentication.yaml
       ├── monitoring.coreos.com
       │   └── servicemonitors.yaml
       ├── networking.k8s.io
       │   └── networkpolicies.yaml
       ├── openshift-keda.yaml
       ├── pods
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-ptgwx
       │   │   ├── custom-metrics-autoscaler-operator
       │   │   │   └── custom-metrics-autoscaler-operator
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── custom-metrics-autoscaler-operator-58bd9f458-ptgwx.yaml
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-thbsh
       │   │   └── custom-metrics-autoscaler-operator
       │   │       └── custom-metrics-autoscaler-operator
       │   │           └── logs
       │   ├── keda-metrics-apiserver-65c7cc44fd-6wq4g
       │   │   ├── keda-metrics-apiserver
       │   │   │   └── keda-metrics-apiserver
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── keda-metrics-apiserver-65c7cc44fd-6wq4g.yaml
       │   └── keda-operator-776cbb6768-fb6m5
       │       ├── keda-operator
       │       │   └── keda-operator
       │       │       └── logs
       │       │           ├── current.log
       │       │           ├── previous.insecure.log
       │       │           └── previous.log
       │       └── keda-operator-776cbb6768-fb6m5.yaml
       ├── policy
       │   └── poddisruptionbudgets.yaml
       └── route.openshift.io
           └── routes.yaml

3. Create a compressed file from the

   must-gather

   directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 

   1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Select the Administrator perspective in the OpenShift Container Platform web console.
2. Select Observe Metrics.
3. To create a custom query, add your PromQL query to the Expression field.
4. To add multiple queries, select Add Query.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. Switch to the openshift-keda project.

3. Remove the

   KedaController

   custom resource.
   1. Find the CustomMetricsAutoscaler Operator and click the KedaController tab.
   2. Find the custom resource, and then click Delete KedaController.
   3. Click Uninstall.

4. Remove the Custom Metrics Autoscaler Operator:

   1. Click Operators Installed Operators.
   2. Find the CustomMetricsAutoscaler Operator and click the Options menu and select Uninstall Operator.
   3. Click Uninstall.

5. Optional: Use the OpenShift CLI to remove the custom metrics autoscaler components:

   1. Delete the custom metrics autoscaler CRDs:

      • clustertriggerauthentications.keda.sh

      • kedacontrollers.keda.sh

      • scaledjobs.keda.sh

      • scaledobjects.keda.sh

      • triggerauthentications.keda.sh

      $ oc delete crd clustertriggerauthentications.keda.sh kedacontrollers.keda.sh scaledjobs.keda.sh scaledobjects.keda.sh triggerauthentications.keda.sh

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings. However, there might be a few cluster roles that must be manually deleted.

   2. List any custom metrics autoscaler cluster roles:

      $ oc get clusterrole | grep keda.sh

   3. Delete the listed custom metrics autoscaler cluster roles. For example:

      $ oc delete clusterrole.keda.sh-v1alpha1-admin

   4. List any custom metrics autoscaler cluster role bindings:

      $ oc get clusterrolebinding | grep keda.sh

   5. Delete the listed custom metrics autoscaler cluster role bindings. For example:

      $ oc delete clusterrolebinding.keda.sh-v1alpha1-admin

6. Delete the custom metrics autoscaler project:

   $ oc delete project openshift-keda

7. Delete the Cluster Metric Autoscaler Operator:

   $ oc delete operator/openshift-custom-metrics-autoscaler-operator.openshift-keda