Nodes

1. Change to the project:

   $ oc project <project-name>

2. Run the following command:

   $ oc get pods

   For example:

   $ oc get pods

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   console-698d866b78-bnshf   1/1     Running   2          165m
   console-698d866b78-m87pm   1/1     Running   2          165m

   Add the -o wide flags to view the pod IP address and the node where the pod is located.

   $ oc get pods -o wide

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                           NOMINATED NODE
   console-698d866b78-bnshf   1/1     Running   2          166m   10.128.0.24   ip-10-0-152-71.ec2.internal    <none>
   console-698d866b78-m87pm   1/1     Running   2          166m   10.129.0.23   ip-10-0-173-237.ec2.internal   <none>

1. Run the following command:

   $ oc adm top pods

   For example:

   $ oc adm top pods -n openshift-console

   Example output

   NAME                         CPU(cores)   MEMORY(bytes)
   console-7f58c69899-q8c8k     0m           22Mi
   console-7f58c69899-xhbgg     0m           25Mi
   downloads-594fcccf94-bcxk8   3m           18Mi
   downloads-594fcccf94-kv4p6   2m           15Mi

2. Run the following command to view the usage statistics for pods with labels:

   $ oc adm top pod --selector=''

   You must choose the selector (label query) to filter on. Supports =, ==, and !=.

   For example:

   $ oc adm top pod --selector='name=my-pod'

Procedure (UI)

1. In the OpenShift Container Platform console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.

   Note

   Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.

2. Select a project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs.

1. Write an object definition JSON file, and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod object definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

1. Create a Pod spec or edit existing pods to include the system-cluster-critical priority class:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pdb
   spec:
     template:
       metadata:
         name: critical-pod
       priorityClassName: system-cluster-critical 1

   1

   Default priority class for pods that should never be evicted from a node.

   Alternatively, you can specify system-node-critical for pods that are important to the cluster but can be removed if necessary.

2. Create the pod:

   $ oc create -f <file-name>.yaml

1. In the Topology view, click the node to reveal the side pane.

2. From the Actions drop-down list, select Add HorizontalPodAutoscaler to open the Add HorizontalPodAutoscaler form.

   Figure 2.2. Add HorizontalPodAutoscaler

   

3. From the Add HorizontalPodAutoscaler form, define the name, minimum and maximum pod limits, the CPU and memory usage, and click Save.

   Note

   If any of the values for CPU and memory usage are missing, a warning is displayed.

1. In the Topology view, click the node to reveal the side pane.
2. From the Actions drop-down list, select Edit HorizontalPodAutoscaler to open the Edit Horizontal Pod Autoscaler form.
3. From the Edit Horizontal Pod Autoscaler form, edit the minimum and maximum pod limits and the CPU and memory usage, and click Save.

1. In the Topology view, click the node to reveal the side panel.
2. From the Actions drop-down list, select Remove HorizontalPodAutoscaler.
3. In the confirmation pop-up window, click Remove to remove the HPA.

1. Perform one of the following:

   • To scale based on the percent of CPU utilization, create a HorizontalPodAutoscaler object for an existing object:

     $ oc autoscale <object_type>/<name> \1
       --min <number> \2
       --max <number> \3
       --cpu-percent=<percent> 4

     1

     Specify the type and name of the object to autoscale. The object must exist and be a Deployment, DeploymentConfig/dc, ReplicaSet/rs, ReplicationController/rc, or StatefulSet.

     2

     Optionally, specify the minimum number of replicas when scaling down.

     3

     Specify the maximum number of replicas when scaling up.

     4

     Specify the target average CPU utilization over all the pods, represented as a percent of requested CPU. If not specified or negative, a default autoscaling policy is used.

     For example, the following command shows autoscaling for the image-registry Deployment object. The initial deployment requires 3 pods. The HPA object increases the minimum to 5. If CPU usage on the pods reaches 75%, the pods will increase to 7:

     $ oc autoscale deployment/image-registry --min=5 --max=7 --cpu-percent=75

   • To scale for a specific CPU value, create a YAML file similar to the following for an existing object:

     1. Create a YAML file similar to the following:

        apiVersion: autoscaling/v2 1
        kind: HorizontalPodAutoscaler
        metadata:
          name: cpu-autoscale 2
          namespace: default
        spec:
          scaleTargetRef:
            apiVersion: apps/v1 3
            kind: Deployment 4
            name: example 5
          minReplicas: 1 6
          maxReplicas: 10 7
          metrics: 8
          - type: Resource
            resource:
              name: cpu 9
              target:
                type: AverageValue 10
                averageValue: 500m 11

        1

        Use the autoscaling/v2 API.

        2

        Specify a name for this horizontal pod autoscaler object.

        3

        Specify the API version of the object to scale:

        • For a Deployment, ReplicaSet, Statefulset object, use apps/v1.
        • For a ReplicationController, use v1.
        • For a DeploymentConfig, use apps.openshift.io/v1.

        4

        Specify the type of object. The object must be a Deployment, DeploymentConfig/dc, ReplicaSet/rs, ReplicationController/rc, or StatefulSet.

        5

        Specify the name of the object to scale. The object must exist.

        6

        Specify the minimum number of replicas when scaling down.

        7

        Specify the maximum number of replicas when scaling up.

        8

        Use the metrics parameter for memory utilization.

        9

        Specify cpu for CPU utilization.

        10

        Set to AverageValue.

        11

        Set to averageValue with the targeted CPU value.

     2. Create the horizontal pod autoscaler:

        $ oc create -f <file-name>.yaml

2. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa cpu-autoscale

   Example output

   NAME            REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   cpu-autoscale   Deployment/example   173m/500m       1         10        1          20m

1. Create a YAML file for one of the following:

   • To scale for a specific memory value, create a HorizontalPodAutoscaler object similar to the following for an existing object:

     apiVersion: autoscaling/v2 1
     kind: HorizontalPodAutoscaler
     metadata:
       name: hpa-resource-metrics-memory 2
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: apps/v1 3
         kind: Deployment 4
         name: example 5
       minReplicas: 1 6
       maxReplicas: 10 7
       metrics: 8
       - type: Resource
         resource:
           name: memory 9
           target:
             type: AverageValue 10
             averageValue: 500Mi 11
       behavior: 12
         scaleDown:
           stabilizationWindowSeconds: 300
           policies:
           - type: Pods
             value: 4
             periodSeconds: 60
           - type: Percent
             value: 10
             periodSeconds: 60
           selectPolicy: Max

     1

     Use the autoscaling/v2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a Deployment, ReplicaSet, or Statefulset object, use apps/v1.
     • For a ReplicationController, use v1.
     • For a DeploymentConfig, use apps.openshift.io/v1.

     4

     Specify the type of object. The object must be a Deployment, DeploymentConfig, ReplicaSet, ReplicationController, or StatefulSet.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set the type to AverageValue.

     11

     Specify averageValue and a specific memory value.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

   • To scale for a percentage, create a HorizontalPodAutoscaler object similar to the following for an existing object:

     apiVersion: autoscaling/v2 1
     kind: HorizontalPodAutoscaler
     metadata:
       name: memory-autoscale 2
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: apps/v1 3
         kind: Deployment 4
         name: example 5
       minReplicas: 1 6
       maxReplicas: 10 7
       metrics: 8
       - type: Resource
         resource:
           name: memory 9
           target:
             type: Utilization 10
             averageUtilization: 50 11
       behavior: 12
         scaleUp:
           stabilizationWindowSeconds: 180
           policies:
           - type: Pods
             value: 6
             periodSeconds: 120
           - type: Percent
             value: 10
             periodSeconds: 120
           selectPolicy: Max

     1

     Use the autoscaling/v2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a ReplicationController, use v1.
     • For a DeploymentConfig, use apps.openshift.io/v1.
     • For a Deployment, ReplicaSet, Statefulset object, use apps/v1.

     4

     Specify the type of object. The object must be a Deployment, DeploymentConfig, ReplicaSet, ReplicationController, or StatefulSet.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set to Utilization.

     11

     Specify averageUtilization and a target average memory utilization over all the pods, represented as a percent of requested memory. The target pods must have memory requests configured.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

2. Create the horizontal pod autoscaler:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f hpa.yaml

   Example output

   horizontalpodautoscaler.autoscaling/hpa-resource-metrics-memory created

3. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa hpa-resource-metrics-memory

   Example output

   NAME                          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   hpa-resource-metrics-memory   Deployment/example   2441216/500Mi   1         10        1          20m

   $ oc describe hpa hpa-resource-metrics-memory

   Example output

   Name:                        hpa-resource-metrics-memory
   Namespace:                   default
   Labels:                      <none>
   Annotations:                 <none>
   CreationTimestamp:           Wed, 04 Mar 2020 16:31:37 +0530
   Reference:                   Deployment/example
   Metrics:                     ( current / target )
     resource memory on pods:   2441216 / 500Mi
   Min replicas:                1
   Max replicas:                10
   ReplicationController pods:  1 current / 1 desired
   Conditions:
     Type            Status  Reason              Message
     ----            ------  ------              -------
     AbleToScale     True    ReadyForNewScale    recommended size matches current size
     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
   Events:
     Type     Reason                   Age                 From                       Message
     ----     ------                   ----                ----                       -------
     Normal   SuccessfulRescale        6m34s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose VerticalPodAutoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory openshift-vertical-pod-autoscaler namespace, which is automatically created if it does not exist.
4. Click Install.

Verifiction

1. Verify the installation by listing the VPA Operator components:

   1. Navigate to Workloads → Pods.
   2. Select the openshift-vertical-pod-autoscaler project from the drop-down menu and verify that there are four pods running.
   3. Navigate to Workloads → Deployments to verify that there are four deployments running.

2. Optional: Verify the installation in the OpenShift Container Platform CLI using the following command:

   $ oc get all -n openshift-vertical-pod-autoscaler

   The output shows four pods and four deplyoments:

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/vertical-pod-autoscaler-operator-85b4569c47-2gmhc   1/1     Running   0          3m13s
   pod/vpa-admission-plugin-default-67644fc87f-xq7k9       1/1     Running   0          2m56s
   pod/vpa-recommender-default-7c54764b59-8gckt            1/1     Running   0          2m56s
   pod/vpa-updater-default-7f6cc87858-47vw9                1/1     Running   0          2m56s
   
   NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   service/vpa-webhook   ClusterIP   172.30.53.206   <none>        443/TCP   2m56s
   
   NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/vertical-pod-autoscaler-operator   1/1     1            1           3m13s
   deployment.apps/vpa-admission-plugin-default       1/1     1            1           2m56s
   deployment.apps/vpa-recommender-default            1/1     1            1           2m56s
   deployment.apps/vpa-updater-default                1/1     1            1           2m56s
   
   NAME                                                          DESIRED   CURRENT   READY   AGE
   replicaset.apps/vertical-pod-autoscaler-operator-85b4569c47   1         1         1       3m13s
   replicaset.apps/vpa-admission-plugin-default-67644fc87f       1         1         1       2m56s
   replicaset.apps/vpa-recommender-default-7c54764b59            1         1         1       2m56s
   replicaset.apps/vpa-updater-default-7f6cc87858                1         1         1       2m56s

1. Change to the project where the workload object you want to scale is located.

   1. Create a VPA CR YAML file:

      apiVersion: autoscaling.k8s.io/v1
      kind: VerticalPodAutoscaler
      metadata:
        name: vpa-recommender
      spec:
        targetRef:
          apiVersion: "apps/v1"
          kind:       Deployment 1
          name:       frontend 2
        updatePolicy:
          updateMode: "Auto" 3
        resourcePolicy: 4
          containerPolicies:
          - containerName: my-opt-sidecar
            mode: "Off"
        recommenders: 5
          - name: my-recommender

      1

      Specify the type of workload object you want this VPA to manage: Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, or ReplicationController.

      2

      Specify the name of an existing workload object you want this VPA to manage.

      3

      Specify the VPA mode:

      • auto to automatically apply the recommended resources on pods associated with the controller. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests.
      • recreate to automatically apply the recommended resources on pods associated with the workload object. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests. The recreate mode should be used rarely, only if you need to ensure that the pods are restarted whenever the resource request changes.
      • initial to automatically apply the recommended resources when pods associated with the workload object are created. The VPA does not update the pods as it learns new resource recommendations.
      • off to only generate resource recommendations for the pods associated with the workload object. The VPA does not update the pods as it learns new resource recommendations and does not apply the recommendations to new pods.

      4

      Optional. Specify the containers you want to opt-out and set the mode to Off.

      5

      Optional. Specify an alternative recommender.

   2. Create the VPA CR:

      $ oc create -f <file-name>.yaml

      After a few moments, the VPA learns the resource usage of the containers in the pods associated with the workload object.

      You can view the VPA recommendations using the following command:

      $ oc get vpa <vpa-name> --output yaml

      The output shows the recommendations for CPU and memory requests, similar to the following:

      Example output

      ...
      status:
      
      ...
      
        recommendation:
          containerRecommendations:
          - containerName: frontend
            lowerBound: 1
              cpu: 25m
              memory: 262144k
            target: 2
              cpu: 25m
              memory: 262144k
            uncappedTarget: 3
              cpu: 25m
              memory: 262144k
            upperBound: 4
              cpu: 262m
              memory: "274357142"
          - containerName: backend
            lowerBound:
              cpu: 12m
              memory: 131072k
            target:
              cpu: 12m
              memory: 131072k
            uncappedTarget:
              cpu: 12m
              memory: 131072k
            upperBound:
              cpu: 476m
              memory: "498558823"
      
      ...

      1

      lowerBound is the minimum recommended resource levels.

      2

      target is the recommended resource levels.

      3

      upperBound is the highest recommended resource levels.

      4

      uncappedTarget is the most recent resource recommendations.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
2. Switch to the openshift-vertical-pod-autoscaler project.
3. For the VerticalPodAutoscaler Operator, click the Options menu and select Uninstall Operator.
4. Optional: To remove all operands associated with the Operator, in the dialog box, select Delete all operand instances for this operator checkbox.
5. Click Uninstall.

6. Optional: Use the OpenShift CLI to remove the VPA components:

   1. Delete the VPA namespace:

      $ oc delete namespace openshift-vertical-pod-autoscaler

   2. Delete the VPA custom resource definition (CRD) objects:

      $ oc delete crd verticalpodautoscalercheckpoints.autoscaling.k8s.io

      $ oc delete crd verticalpodautoscalercontrollers.autoscaling.openshift.io

      $ oc delete crd verticalpodautoscalers.autoscaling.k8s.io

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings.

      Note

      This action removes from the cluster all user-created VPA CRs. If you re-install the VPA, you must create these objects again.

   3. Delete the MutatingWebhookConfiguration object by running the following command:

      $ oc delete MutatingWebhookConfiguration vpa-webhook-config

   4. Delete the VPA Operator:

      $ oc delete operator/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler

1. Create a secret object that contains the data you want to keep secret. The specific data required for each secret type is descibed in the following sections.

   Example YAML object that creates an opaque secret

   apiVersion: v1
   kind: Secret
   metadata:
     name: test-secret
   type: Opaque 1
   data: 2
     username: <username>
     password: <password>
   stringData: 3
     hostname: myapp.mydomain.com
     secret.properties: |
       property1=valueA
       property2=valueB

   1

   Specifies the type of secret.

   2

   Specifies encoded string and data.

   3

   Specifies decoded string and data.

   Use either the data or stringdata fields, not both.

2. Update the pod’s service account to reference the secret:

   YAML of a service account that uses a secret

   apiVersion: v1
   kind: ServiceAccount
    ...
   secrets:
   - name: test-secret

3. Create a pod, which consumes the secret as an environment variable or as a file (using a secret volume):

   YAML of a pod populating files in a volume with secret data

   apiVersion: v1
   kind: Pod
   metadata:
     name: secret-example-pod
   spec:
     containers:
       - name: secret-test-container
         image: busybox
         command: [ "/bin/sh", "-c", "cat /etc/secret-volume/*" ]
         volumeMounts: 1
             - name: secret-volume
               mountPath: /etc/secret-volume 2
               readOnly: true 3
     volumes:
       - name: secret-volume
         secret:
           secretName: test-secret 4
     restartPolicy: Never

   1

   Add a volumeMounts field to each container that needs the secret.

   2

   Specifies an unused directory name where you would like the secret to appear. Each key in the secret data map becomes the filename under mountPath.

   3

   Set to true. If true, this instructs the driver to provide a read-only volume.

   4

   Specifies the name of the secret.

   YAML of a pod populating environment variables with secret data

   apiVersion: v1
   kind: Pod
   metadata:
     name: secret-example-pod
   spec:
     containers:
       - name: secret-test-container
         image: busybox
         command: [ "/bin/sh", "-c", "export" ]
         env:
           - name: TEST_SECRET_USERNAME_ENV_VAR
             valueFrom:
               secretKeyRef: 1
                 name: test-secret
                 key: username
     restartPolicy: Never

   1

   Specifies the environment variable that consumes the secret key.

   YAML of a build config populating environment variables with secret data

   apiVersion: build.openshift.io/v1
   kind: BuildConfig
   metadata:
     name: secret-example-bc
   spec:
     strategy:
       sourceStrategy:
         env:
         - name: TEST_SECRET_USERNAME_ENV_VAR
           valueFrom:
             secretKeyRef: 1
               name: test-secret
               key: username
         from:
           kind: ImageStreamTag
           namespace: openshift
           name: 'cli:latest'

   1

   Specifies the environment variable that consumes the secret key.

Procedure

1. Create a service account in your namespace by running the following command:

   $ oc create sa <service_account_name> -n <your_namespace>

2. Save the following YAML example to a file named service-account-token-secret.yaml. The example includes a Secret object configuration that you can use to generate a service account token:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name> 1
     annotations:
       kubernetes.io/service-account.name: "sa-name" 2
   type: kubernetes.io/service-account-token 3

   1

   Replace <secret_name> with the name of your service token secret.

   2

   Specifies an existing service account name. If you are creating both the ServiceAccount and the Secret objects, create the ServiceAccount object first.

   3

   Specifies a service account token secret type.

3. Generate the service account token by applying the file:

   $ oc apply -f service-account-token-secret.yaml

4. Get the service account token from the secret by running the following command:

   $ oc get secret <sa_token_secret> -o jsonpath='{.data.token}' | base64 --decode 1

   Example output

   ayJhbGciOiJSUzI1NiIsImtpZCI6IklOb2dtck1qZ3hCSWpoNnh5YnZhSE9QMkk3YnRZMVZoclFfQTZfRFp1YlUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImJ1aWxkZXItdG9rZW4tdHZrbnIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYnVpbGRlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNmZGU2MGZmLTA1NGYtNDkyZi04YzhjLTNlZjE0NDk3MmFmNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmJ1aWxkZXIifQ.OmqFTDuMHC_lYvvEUrjr1x453hlEEHYcxS9VKSzmRkP1SiVZWPNPkTWlfNRp6bIUZD3U6aN3N7dMSN0eI5hu36xPgpKTdvuckKLTCnelMx6cxOdAbrcw1mCmOClNscwjS1KO1kzMtYnnq8rXHiMJELsNlhnRyyIXRTtNBsy4t64T3283s3SLsancyx0gy0ujx-Ch3uKAKdZi5iT-I8jnnQ-ds5THDs2h65RJhgglQEmSxpHrLGZFmyHAQI-_SjvmHZPXEc482x3SkaQHNLqpmrpJorNqh1M8ZHKzlujhZgVooMvJmWPXTb2vnvi3DGn2XI-hZxl1yD2yGH1RBpYUHA

   1

   Replace <sa_token_secret> with the name of your service token secret.

5. Use your service account token to authenticate with the API of your cluster:

   $ curl -X GET <openshift_cluster_api> --header "Authorization: Bearer <token>" 1 2

   1

   Replace <openshift_cluster_api> with the OpenShift cluster API.

   2

   Replace <token> with the service account token that is output in the preceding command.

1. Delete the secret:

   $ oc delete secret <secret_name>

2. Clear the annotations:

   $ oc annotate service <service_name> service.beta.openshift.io/serving-cert-generation-error-

   $ oc annotate service <service_name> service.beta.openshift.io/serving-cert-generation-error-num-

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command. Perform one of the following steps:

   1. View the machine config:

      # oc describe machineconfig <name>

      For example:

      # oc describe machineconfig 00-worker

      Example output

      Name:         00-worker
      Namespace:
      Labels:       machineconfiguration.openshift.io/role=worker 1

      1

      Label required for the Device Manager.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a Device Manager CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: devicemgr 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
          machineconfiguration.openshift.io: devicemgr 2
     kubeletConfig:
       feature-gates:
         - DevicePlugins=true 3

   1

   Assign a name to CR.

   2

   Enter the label from the Machine Config Pool.

   3

   Set DevicePlugins to 'true`.

2. Create the Device Manager:

   $ oc create -f devicemgr.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/devicemgr created

3. Ensure that Device Manager was actually enabled by confirming that /var/lib/kubelet/device-plugins/kubelet.sock is created on the node. This is the UNIX domain socket on which the Device Manager gRPC server listens for new plugin registrations. This sock file is created when the Kubelet is started only if Device Manager is enabled.

1. Create one or more priority classes:

   1. Create a YAML file similar to the following:

      apiVersion: scheduling.k8s.io/v1
      kind: PriorityClass
      metadata:
        name: high-priority 1
      value: 1000000 2
      preemptionPolicy: PreemptLowerPriority 3
      globalDefault: false 4
      description: "This priority class should be used for XYZ service pods only." 5

      1

      The name of the priority class object.

      2

      The priority value of the object.

      3

      Optional. Specifies whether this priority class is preempting or non-preempting. The preemption policy defaults to PreemptLowerPriority, which allows pods of that priority class to preempt lower-priority pods. If the preemption policy is set to Never, pods in that priority class are non-preempting.

      4

      Optional. Specifies whether this priority class should be used for pods without a priority class name specified. This field is false by default. Only one priority class with globalDefault set to true can exist in the cluster. If there is no priority class with globalDefault:true, the priority of pods with no priority class name is zero. Adding a priority class with globalDefault:true affects only pods created after the priority class is added and does not change the priorities of existing pods.

      5

      Optional. Describes which pods developers should use with this priority class. Enter an arbitrary text string.

   2. Create the priority class:

      $ oc create -f <file-name>.yaml

2. Create a pod spec to include the name of a priority class:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx
        labels:
          env: test
      spec:
        containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
        priorityClassName: high-priority 1

      1

      Specify the priority class to use with this pod.

   2. Create the pod:

      $ oc create -f <file-name>.yaml

      You can add the priority name directly to the pod configuration or to a pod template.

Procedure

1. Add labels to a node by using a compute machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the compute machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet abc612-msrtw-worker-us-east-1c  --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

        Tip

        You can alternatively apply the following YAML to add labels to a compute machine set:

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
          name: xf2bd-infra-us-east-2a
          namespace: openshift-machine-api
        spec:
          template:
            spec:
              metadata:
                labels:
                  region: "east"
                  type: "user-node"
        #...

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        
        # ...
        
        spec:
        # ...
          template:
            metadata:
        # ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
        # ...

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east

        Tip

        You can alternatively apply the following YAML to add labels to a node:

        kind: Node
        apiVersion: v1
        metadata:
          name: hello-node-6fbccf8d9
          labels:
            type: "user-node"
            region: "east"
        #...

     2. Verify that the labels are added to the node:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                          STATUS   ROLES    AGE   VERSION
        ip-10-0-142-25.ec2.internal   Ready    worker   17m   v1.25.0

2. Add the matching node selector to a pod:

   • To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:

     Example ReplicaSet object with labels

     kind: ReplicaSet
     apiVersion: apps/v1
     metadata:
       name: hello-node-6fbccf8d9
     # ...
     spec:
     # ...
       template:
         metadata:
           creationTimestamp: null
           labels:
             ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
             pod-template-hash: 66d5cf9464
         spec:
           nodeSelector:
             kubernetes.io/os: linux
             node-role.kubernetes.io/worker: ''
             type: user-node 1
     #...

     1

     Add the node selector.

   • To add a node selector to a specific, new pod, add the selector to the Pod object directly:

     Example Pod object with a node selector

     apiVersion: v1
     kind: Pod
     metadata:
       name: hello-node-6fbccf8d9
     #...
     spec:
       nodeSelector:
         region: east
         type: user-node
     #...

     Note

     You cannot add a node selector directly to an existing scheduled pod.

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Custom Metrics Autoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the All namespaces on the cluster (default) option is selected for Installation Mode. This installs the Operator in all namespaces.
4. Ensure that the openshift-keda namespace is selected for Installed Namespace. OpenShift Container Platform creates the namespace, if not present in your cluster.
5. Click Install.

6. Verify the installation by listing the Custom Metrics Autoscaler Operator components:

   1. Navigate to Workloads → Pods.
   2. Select the openshift-keda project from the drop-down menu and verify that the custom-metrics-autoscaler-operator-* pod is running.
   3. Navigate to Workloads → Deployments to verify that the custom-metrics-autoscaler-operator deployment is running.

7. Optional: Verify the installation in the OpenShift CLI using the following commands:

   $ oc get all -n openshift-keda

   The output appears similar to the following:

   Example output

   NAME                                                      READY   STATUS    RESTARTS   AGE
   pod/custom-metrics-autoscaler-operator-5fd8d9ffd8-xt4xp   1/1     Running   0          18m
   
   NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/custom-metrics-autoscaler-operator   1/1     1            1           18m
   
   NAME                                                            DESIRED   CURRENT   READY   AGE
   replicaset.apps/custom-metrics-autoscaler-operator-5fd8d9ffd8   1         1         1       18m

8. Install the KedaController custom resource, which creates the required CRDs:

   1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
   2. Click Custom Metrics Autoscaler.
   3. On the Operator Details page, click the KedaController tab.

   4. On the KedaController tab, click Create KedaController and edit the file.

      kind: KedaController
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: keda
        namespace: openshift-keda
      spec:
        watchNamespace: '' 1
        operator:
          logLevel: info 2
          logEncoder: console 3
          caConfigMaps: 4
          - thanos-cert
          - kafka-cert
        metricsServer:
          logLevel: '0' 5
          auditConfig: 6
            logFormat: "json"
            logOutputVolumeClaim: "persistentVolumeClaimName"
            policy:
              rules:
              - level: Metadata
              omitStages: ["RequestReceived"]
              omitManagedFields: false
            lifetime:
              maxAge: "2"
              maxBackup: "1"
              maxSize: "50"
        serviceAccount: {}

      1

      Specifies a single namespace in which the Custom Metrics Autoscaler Operator should scale applications. Leave it blank or leave it empty to scale applications in all namespaces. This field should have a namespace or be empty. The default value is empty.

      2

      Specifies the level of verbosity for the Custom Metrics Autoscaler Operator log messages. The allowed values are debug, info, error. The default is info.

      3

      Specifies the logging format for the Custom Metrics Autoscaler Operator log messages. The allowed values are console or json. The default is console.

      4

      Optional: Specifies one or more config maps with CA certificates, which the Custom Metrics Autoscaler Operator can use to connect securely to TLS-enabled metrics sources.

      5

      Specifies the logging level for the Custom Metrics Autoscaler Metrics Server. The allowed values are 0 for info and 4 or debug. The default is 0.

      6

      Activates audit logging for the Custom Metrics Autoscaler Operator and specifies the audit policy to use, as described in the "Configuring audit logging" section.

   5. Click Create to create the KEDA controller.

Procedure

1. Create the TriggerAuthentication or ClusterTriggerAuthentication object.

   1. Create a YAML file that defines the object:

      Example trigger authentication with a secret

      kind: TriggerAuthentication
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: prom-triggerauthentication
        namespace: my-namespace
      spec:
        secretTargetRef:
        - parameter: user-name
          name: my-secret
          key: USER_NAME
        - parameter: password
          name: my-secret
          key: USER_PASSWORD

   2. Create the TriggerAuthentication object:

      $ oc create -f <filename>.yaml

2. Create or edit a ScaledObject YAML file that uses the trigger authentication:

   1. Create a YAML file that defines the object by running the following command:

      Example scaled object with a trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-triggerauthentication 1
            kind: TriggerAuthentication 2

      1

      Specify the name of your trigger authentication object.

      2

      Specify TriggerAuthentication. TriggerAuthentication is the default.

      Example scaled object with a cluster trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-cluster-triggerauthentication 1
            kind: ClusterTriggerAuthentication 2

      1

      Specify the name of your trigger authentication object.

      2

      Specify ClusterTriggerAuthentication.

   2. Create the scaled object by running the following command:

      $ oc apply -f <filename>

Procedure

1. Use the following command to edit the ScaledObject CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Add the autoscaling.keda.sh/paused-replicas annotation with any value:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 1
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling.

Procedure

1. Use the following command to edit the ScaledObject CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Remove the autoscaling.keda.sh/paused-replicas annotation.

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 1
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Remove this annotation to restart a paused custom metrics autoscaler.

Procedure

1. Edit the KedaController custom resource to add the auditConfig stanza:

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: openshift-keda
   spec:
   # ...
     metricsServer:
   # ...
       auditConfig:
         logFormat: "json" 1
         logOutputVolumeClaim: "pvc-audit-log" 2
         policy:
           rules: 3
           - level: Metadata
           omitStages: "RequestReceived" 4
           omitManagedFields: false 5
         lifetime: 6
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"

   1

   Specifies the output format of the audit log, either legacy or json.

   2

   Specifies an existing persistent volume claim for storing the log data. All requests coming to the API server are logged to this persistent volume claim. If you leave this field empty, the log data is sent to stdout.

   3

   Specifies which events should be recorded and what data they should include:

   • None: Do not log events.
   • Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. This is the default.
   • Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.
   • RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.

   4

   Specifies stages for which no event is created.

   5

   Specifies whether to omit the managed fields of the request and response bodies from being written to the API audit log, either true to omit the fields or false to include the fields.

   6

   Specifies the size and lifespan of the audit logs.

   • maxAge: The maximum number of days to retain audit log files, based on the timestamp encoded in their filename.
   • maxBackup: The maximum number of audit log files to retain. Set to 0 to retain all audit log files.
   • maxSize: The maximum size in megabytes of an audit log file before it gets rotated.

Verification

1. View the audit log file directly:

   1. Obtain the name of the keda-metrics-apiserver-* pod:

      oc get pod -n openshift-keda

      Example output

      NAME                                                  READY   STATUS    RESTARTS   AGE
      custom-metrics-autoscaler-operator-5cb44cd75d-9v4lv   1/1     Running   0          8m20s
      keda-metrics-apiserver-65c7cc44fd-rrl4r               1/1     Running   0          2m55s
      keda-operator-776cbb6768-zpj5b                        1/1     Running   0          2m55s

   2. View the log data by using a command similar to the following:

      $ oc logs keda-metrics-apiserver-<hash>|grep -i metadata 1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      $ oc logs keda-metrics-apiserver-65c7cc44fd-rrl4r|grep -i metadata

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4c81d41b-3dab-4675-90ce-20b87ce24013","stage":"ResponseComplete","requestURI":"/healthz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.131.0.1"],"userAgent":"kube-probe/1.26","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-02-16T13:00:03.554567Z","stageTimestamp":"2023-02-16T13:00:03.555032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
       ...

2. Alternatively, you can view a specific log:

   1. Use a command similar to the following to log into the keda-metrics-apiserver-* pod:

      $ oc rsh pod/keda-metrics-apiserver-<hash> -n openshift-keda

      For example:

      $ oc rsh pod/keda-metrics-apiserver-65c7cc44fd-rrl4r -n openshift-keda

   2. Change to the /var/audit-policy/ directory:

      sh-4.4$ cd /var/audit-policy/

   3. List the available logs:

      sh-4.4$ ls

      Example output

      log-2023.02.17-14:50  policy.yaml

   4. View the log, as needed:

      sh-4.4$ cat <log_name>/<pvc_name>|grep -i <log_level> 1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      sh-4.4$ cat log-2023.02.17-14:50/pvc-audit-log|grep -i Request

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"63e7f68c-04ec-4f4d-8749-bf1656572a41","stage":"ResponseComplete","requestURI":"/openapi/v2","verb":"get","user":{"username":"system:aggregator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"responseStatus":{"metadata":{},"code":304},"requestReceivedTimestamp":"2023-02-17T13:12:55.035478Z","stageTimestamp":"2023-02-17T13:12:55.038346Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
       ...

Procedure

1. Navigate to the directory where you want to store the must-gather data.

   Note

   If your cluster is using a restricted network, you must take additional steps. If your mirror registry has a trusted CA, you must first add the trusted CA to the cluster. For all clusters on restricted networks, you must import the default must-gather image as an image stream by running the following command.

   $ oc import-image is/must-gather -n openshift

2. Perform one of the following:

   • To get only the Custom Metrics Autoscaler Operator must-gather data, use the following command:

     $ oc adm must-gather --image="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
     -n openshift-marketplace \
     -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     The custom image for the must-gather command is pulled directly from the Operator package manifests, so that it works on any cluster where the Custom Metric Autoscaler Operator is available.

   • To gather the default must-gather data in addition to the Custom Metric Autoscaler Operator information:

     1. Use the following command to obtain the Custom Metrics Autoscaler Operator image and set it as an environment variable:

        $ IMAGE="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
          -n openshift-marketplace \
          -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     2. Use the oc adm must-gather with the Custom Metrics Autoscaler Operator image:

        $ oc adm must-gather --image-stream=openshift/must-gather --image=${IMAGE}

   Example 3.1. Example must-gather output for the Custom Metric Autoscaler:

   └── openshift-keda
       ├── apps
       │   ├── daemonsets.yaml
       │   ├── deployments.yaml
       │   ├── replicasets.yaml
       │   └── statefulsets.yaml
       ├── apps.openshift.io
       │   └── deploymentconfigs.yaml
       ├── autoscaling
       │   └── horizontalpodautoscalers.yaml
       ├── batch
       │   ├── cronjobs.yaml
       │   └── jobs.yaml
       ├── build.openshift.io
       │   ├── buildconfigs.yaml
       │   └── builds.yaml
       ├── core
       │   ├── configmaps.yaml
       │   ├── endpoints.yaml
       │   ├── events.yaml
       │   ├── persistentvolumeclaims.yaml
       │   ├── pods.yaml
       │   ├── replicationcontrollers.yaml
       │   ├── secrets.yaml
       │   └── services.yaml
       ├── discovery.k8s.io
       │   └── endpointslices.yaml
       ├── image.openshift.io
       │   └── imagestreams.yaml
       ├── k8s.ovn.org
       │   ├── egressfirewalls.yaml
       │   └── egressqoses.yaml
       ├── keda.sh
       │   ├── kedacontrollers
       │   │   └── keda.yaml
       │   ├── scaledobjects
       │   │   └── example-scaledobject.yaml
       │   └── triggerauthentications
       │       └── example-triggerauthentication.yaml
       ├── monitoring.coreos.com
       │   └── servicemonitors.yaml
       ├── networking.k8s.io
       │   └── networkpolicies.yaml
       ├── openshift-keda.yaml
       ├── pods
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-ptgwx
       │   │   ├── custom-metrics-autoscaler-operator
       │   │   │   └── custom-metrics-autoscaler-operator
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── custom-metrics-autoscaler-operator-58bd9f458-ptgwx.yaml
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-thbsh
       │   │   └── custom-metrics-autoscaler-operator
       │   │       └── custom-metrics-autoscaler-operator
       │   │           └── logs
       │   ├── keda-metrics-apiserver-65c7cc44fd-6wq4g
       │   │   ├── keda-metrics-apiserver
       │   │   │   └── keda-metrics-apiserver
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── keda-metrics-apiserver-65c7cc44fd-6wq4g.yaml
       │   └── keda-operator-776cbb6768-fb6m5
       │       ├── keda-operator
       │       │   └── keda-operator
       │       │       └── logs
       │       │           ├── current.log
       │       │           ├── previous.insecure.log
       │       │           └── previous.log
       │       └── keda-operator-776cbb6768-fb6m5.yaml
       ├── policy
       │   └── poddisruptionbudgets.yaml
       └── route.openshift.io
           └── routes.yaml

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Select the Administrator perspective in the OpenShift Container Platform web console.
2. Select Observe → Metrics.
3. To create a custom query, add your PromQL query to the Expression field.
4. To add multiple queries, select Add Query.

Procedure

1. Create a YAML file similar to the following. Only the name <2>, object name <4>, and object kind <5> are required:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "0" 1
     name: scaledobject 2
     namespace: my-namespace
   spec:
     scaleTargetRef:
       apiVersion: apps/v1 3
       name: example-deployment 4
       kind: Deployment 5
       envSourceContainerName: .spec.template.spec.containers[0] 6
     cooldownPeriod:  200 7
     maxReplicaCount: 100 8
     minReplicaCount: 0 9
     metricsServer: 10
       auditConfig:
         logFormat: "json"
         logOutputVolumeClaim: "persistentVolumeClaimName"
         policy:
           rules:
           - level: Metadata
           omitStages: "RequestReceived"
           omitManagedFields: false
         lifetime:
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"
     fallback: 11
       failureThreshold: 3
       replicas: 6
     pollingInterval: 30 12
     advanced:
       restoreToOriginalReplicaCount: false 13
       horizontalPodAutoscalerConfig:
         name: keda-hpa-scale-down 14
         behavior: 15
           scaleDown:
             stabilizationWindowSeconds: 300
             policies:
             - type: Percent
               value: 100
               periodSeconds: 15
     triggers:
     - type: prometheus 16
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: basic
       authenticationRef: 17
         name: prom-triggerauthentication
         kind: TriggerAuthentication

   1

   Optional: Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling, as described in the "Pausing the custom metrics autoscaler for a workload" section.

   2

   Specifies a name for this custom metrics autoscaler.

   3

   Optional: Specifies the API version of the target resource. The default is apps/v1.

   4

   Specifies the name of the object that you want to scale.

   5

   Specifies the kind as Deployment, StatefulSet or CustomResource.

   6

   Optional: Specifies the name of the container in the target resource, from which the custom metrics autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].

   7

   Optional. Specifies the period in seconds to wait after the last trigger is reported before scaling the deployment back to 0 if the minReplicaCount is set to 0. The default is 300.

   8

   Optional: Specifies the maximum number of replicas when scaling up. The default is 100.

   9

   Optional: Specifies the minimum number of replicas when scaling down.

   10

   Optional: Specifies the parameters for audit logs. as described in the "Configuring audit logging" section.

   11

   Optional: Specifies the number of replicas to fall back to if a scaler fails to get metrics from the source for the number of times defined by the failureThreshold parameter. For more information on fallback behavior, see the KEDA documentation.

   12

   Optional: Specifies the interval in seconds to check each trigger on. The default is 30.

   13

   Optional: Specifies whether to scale back the target resource to the original replica count after the scaled object is deleted. The default is false, which keeps the replica count as it is when the scaled object is deleted.

   14

   Optional: Specifies a name for the horizontal pod autoscaler. The default is keda-hpa-{scaled-object-name}.

   15

   Optional: Specifies a scaling policy to use to control the rate to scale pods up or down, as described in the "Scaling policies" section.

   16

   Specifies the trigger to use as the basis for scaling, as described in the "Understanding the custom metrics autoscaler triggers" section. This example uses OpenShift Container Platform monitoring.

   17

   Optional: Specifies a trigger authentication or a cluster trigger authentication. For more information, see Understanding the custom metrics autoscaler trigger authentication in the Additional resources section.

   • Enter TriggerAuthentication to use a trigger authentication. This is the default.
   • Enter ClusterTriggerAuthentication to use a cluster trigger authentication.

2. Create the custom metrics autoscaler by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. Create a YAML file similar to the following:

   kind: ScaledJob
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: scaledjob
     namespace: my-namespace
   spec:
     failedJobsHistoryLimit: 5
     jobTargetRef:
       activeDeadlineSeconds: 600 1
       backoffLimit: 6 2
       parallelism: 1 3
       completions: 1 4
       template:  5
         metadata:
           name: pi
         spec:
           containers:
           - name: pi
             image: perl
             command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
     maxReplicaCount: 100 6
     pollingInterval: 30 7
     successfulJobsHistoryLimit: 5 8
     failedJobsHistoryLimit: 5 9
     envSourceContainerName: 10
     rolloutStrategy: gradual 11
     scalingStrategy: 12
       strategy: "custom"
       customScalingQueueLengthDeduction: 1
       customScalingRunningJobPercentage: "0.5"
       pendingPodConditions:
         - "Ready"
         - "PodScheduled"
         - "AnyOtherCustomPodCondition"
       multipleScalersCalculation : "max"
     triggers:
     - type: prometheus 13
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: "bearer"
       authenticationRef: 14
         name: prom-cluster-triggerauthentication

   1

   Specifies the maximum duration the job can run.

   2

   Specifies the number of retries for a job. The default is 6.

   3

   Optional: Specifies how many pod replicas a job should run in parallel; defaults to 1.

   • For non-parallel jobs, leave unset. When unset, the default is 1.

   4

   Optional: Specifies how many successful pod completions are needed to mark a job completed.

   • For non-parallel jobs, leave unset. When unset, the default is 1.
   • For parallel jobs with a fixed completion count, specify the number of completions.
   • For parallel jobs with a work queue, leave unset. When unset the default is the value of the parallelism parameter.

   5

   Specifies the template for the pod the controller creates.

   6

   Optional: Specifies the maximum number of replicas when scaling up. The default is 100.

   7

   Optional: Specifies the interval in seconds to check each trigger on. The default is 30.

   8

   Optional: Specifies the number of successful finished jobs should be kept. The default is 100.

   9

   Optional: Specifies how many failed jobs should be kept. The default is 100.

   10

   Optional: Specifies the name of the container in the target resource, from which the custom autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].

   11

   Optional: Specifies whether existing jobs are terminated whenever a scaled job is being updated:

   • default: The autoscaler terminates an existing job if its associated scaled job is updated. The autoscaler recreates the job with the latest specs.
   • gradual: The autoscaler does not terminate an existing job if its associated scaled job is updated. The autoscaler creates new jobs with the latest specs.

   12

   Optional: Specifies a scaling strategy: default, custom, or accurate. The default is default. For more information, see the link in the "Additional resources" section that follows.

   13

   Specifies the trigger to use as the basis for scaling, as described in the "Understanding the custom metrics autoscaler triggers" section.

   14

   Optional: Specifies a trigger authentication or a cluster trigger authentication. For more information, see Understanding the custom metrics autoscaler trigger authentication in the Additional resources section.

   • Enter TriggerAuthentication to use a trigger authentication. This is the default.
   • Enter ClusterTriggerAuthentication to use a cluster trigger authentication.

2. Create the custom metrics autoscaler by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
2. Switch to the openshift-keda project.

3. Remove the KedaController custom resource.

   1. Find the CustomMetricsAutoscaler Operator and click the KedaController tab.
   2. Find the custom resource, and then click Delete KedaController.
   3. Click Uninstall.

4. Remove the Custom Metrics Autoscaler Operator:

   1. Click Operators → Installed Operators.
   2. Find the CustomMetricsAutoscaler Operator and click the Options menu and select Uninstall Operator.
   3. Click Uninstall.

5. Optional: Use the OpenShift CLI to remove the custom metrics autoscaler components:

   1. Delete the custom metrics autoscaler CRDs:

      • clustertriggerauthentications.keda.sh
      • kedacontrollers.keda.sh
      • scaledjobs.keda.sh
      • scaledobjects.keda.sh
      • triggerauthentications.keda.sh

      $ oc delete crd clustertriggerauthentications.keda.sh kedacontrollers.keda.sh scaledjobs.keda.sh scaledobjects.keda.sh triggerauthentications.keda.sh

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings. However, there might be a few cluster roles that must be manually deleted.

   2. List any custom metrics autoscaler cluster roles:

      $ oc get clusterrole | grep keda.sh

   3. Delete the listed custom metrics autoscaler cluster roles. For example:

      $ oc delete clusterrole.keda.sh-v1alpha1-admin

   4. List any custom metrics autoscaler cluster role bindings:

      $ oc get clusterrolebinding | grep keda.sh

   5. Delete the listed custom metrics autoscaler cluster role bindings. For example:

      $ oc delete clusterrolebinding.keda.sh-v1alpha1-admin

6. Delete the custom metrics autoscaler project:

   $ oc delete project openshift-keda

7. Delete the Cluster Metric Autoscaler Operator:

   $ oc delete operator/openshift-custom-metrics-autoscaler-operator.openshift-keda

Procedure

1. Edit the Scheduler object:

   $ oc edit scheduler cluster

2. Specify the profile to use in the spec.profile field:

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   #...
   spec:
     mastersSchedulable: false
     profile: HighNodeUtilization 1
   #...

   1

   Set to LowNodeUtilization, HighNodeUtilization, or NoScoring.

3. Save the file to apply the changes.

Procedure

1. Create a pod with a specific label in the pod spec:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1
        labels:
          security: S1
      spec:
        containers:
        - name: security-s1
          image: docker.io/ocpqe/hello-pod

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

2. When creating other pods, configure the following parameters to add the affinity:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1-east
      #...
      spec
        affinity 1
          podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution: 2
            - labelSelector:
                matchExpressions:
                - key: security 3
                  values:
                  - S1
                  operator: In 4
              topologyKey: topology.kubernetes.io/zone 5
      #...

      1

      Adds a pod affinity.

      2

      Configures the requiredDuringSchedulingIgnoredDuringExecution parameter or the preferredDuringSchedulingIgnoredDuringExecution parameter.

      3

      Specifies the key and values that must be met. If you want the new pod to be scheduled with the other pod, use the same key and values parameters as the label on the first pod.

      4

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

      5

      Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

Procedure

1. Create a pod with a specific label in the pod spec:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1
        labels:
          security: S1
      spec:
        containers:
        - name: security-s1
          image: docker.io/ocpqe/hello-pod

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

2. When creating other pods, configure the following parameters:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s2-east
      #...
      spec
        affinity 1
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 2
            - weight: 100 3
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: security 4
                    values:
                    - S1
                    operator: In 5
                topologyKey: kubernetes.io/hostname 6
      #...

      1

      Adds a pod anti-affinity.

      2

      Configures the requiredDuringSchedulingIgnoredDuringExecution parameter or the preferredDuringSchedulingIgnoredDuringExecution parameter.

      3

      For a preferred rule, specifies a weight for the node, 1-100. The node that with highest weight is preferred.

      4

      Specifies the key and values that must be met. If you want the new pod to not be scheduled with the other pod, use the same key and values parameters as the label on the first pod.

      5

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

      6

      Specifies a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml