Nodes

1. Change to the project:

   $ oc project <project-name>

2. Run the following command:

   $ oc get pods

   For example:

   $ oc get pods

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   console-698d866b78-bnshf   1/1     Running   2          165m
   console-698d866b78-m87pm   1/1     Running   2          165m

   Add the -o wide flags to view the pod IP address and the node where the pod is located.

   $ oc get pods -o wide

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                           NOMINATED NODE
   console-698d866b78-bnshf   1/1     Running   2          166m   10.128.0.24   ip-10-0-152-71.ec2.internal    <none>
   console-698d866b78-m87pm   1/1     Running   2          166m   10.129.0.23   ip-10-0-173-237.ec2.internal   <none>

1. Run the following command:

   $ oc adm top pods

   For example:

   $ oc adm top pods -n openshift-console

   Example output

   NAME                         CPU(cores)   MEMORY(bytes)
   console-7f58c69899-q8c8k     0m           22Mi
   console-7f58c69899-xhbgg     0m           25Mi
   downloads-594fcccf94-bcxk8   3m           18Mi
   downloads-594fcccf94-kv4p6   2m           15Mi

2. Run the following command to view the usage statistics for pods with labels:

   $ oc adm top pod --selector=''

   You must choose the selector (label query) to filter on. Supports =, ==, and !=.

   For example:

   $ oc adm top pod --selector='name=my-pod'

Procedure (UI)

1. In the Red Hat OpenShift Service on AWS console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.

   Note

   Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.

2. Select a project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs.

1. Write an object definition JSON file, and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod object definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

1. Define a pod spec to include the name of a priority class by creating a YAML file similar to the following:

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx
     labels:
       env: test
   spec:
     containers:
     - name: nginx
       image: nginx
       imagePullPolicy: IfNotPresent
     priorityClassName: system-cluster-critical 1

   1

   Specify the priority class to use with this pod.

2. Create the pod:

   $ oc create -f <file-name>.yaml

   You can add the priority name directly to the pod configuration or to a pod template.

Procedure

1. In the Red Hat OpenShift Service on AWS web console, click Operators → OperatorHub.
2. Choose Custom Metrics Autoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the A specific namespace on the cluster option is selected for Installation Mode.
4. For Installed Namespace, click Select a namespace.

5. Click Select Project:

   • If the keda namespace exists, select keda from the list.

   • If the keda namespace does not exist:

     1. Select Create Project to open the Create Project window.
     2. In the Name field, enter keda.
     3. In the Display Name field, enter a descriptive name, such as keda.
     4. Optional: In the Display Name field, add a description for the namespace.
     5. Click Create.
6. Click Install.

7. Verify the installation by listing the Custom Metrics Autoscaler Operator components:

   1. Navigate to Workloads → Pods.
   2. Select the keda project from the drop-down menu and verify that the custom-metrics-autoscaler-operator-* pod is running.
   3. Navigate to Workloads → Deployments to verify that the custom-metrics-autoscaler-operator deployment is running.

8. Optional: Verify the installation in the OpenShift CLI using the following command:

   $ oc get all -n keda

   The output appears similar to the following:

   Example output

   NAME                                                      READY   STATUS    RESTARTS   AGE
   pod/custom-metrics-autoscaler-operator-5fd8d9ffd8-xt4xp   1/1     Running   0          18m
   
   NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/custom-metrics-autoscaler-operator   1/1     1            1           18m
   
   NAME                                                            DESIRED   CURRENT   READY   AGE
   replicaset.apps/custom-metrics-autoscaler-operator-5fd8d9ffd8   1         1         1       18m

9. Install the KedaController custom resource, which creates the required CRDs:

   1. In the Red Hat OpenShift Service on AWS web console, click Operators → Installed Operators.
   2. Click Custom Metrics Autoscaler.
   3. On the Operator Details page, click the KedaController tab.

   4. On the KedaController tab, click Create KedaController and edit the file.

      kind: KedaController
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: keda
        namespace: keda
      spec:
        watchNamespace: '' 1
        operator:
          logLevel: info 2
          logEncoder: console 3
          caConfigMaps: 4
          - thanos-cert
          - kafka-cert
        metricsServer:
          logLevel: '0' 5
          auditConfig: 6
            logFormat: "json"
            logOutputVolumeClaim: "persistentVolumeClaimName"
            policy:
              rules:
              - level: Metadata
              omitStages: ["RequestReceived"]
              omitManagedFields: false
            lifetime:
              maxAge: "2"
              maxBackup: "1"
              maxSize: "50"
        serviceAccount: {}

      1

      Specifies a single namespace in which the Custom Metrics Autoscaler Operator should scale applications. Leave it blank or leave it empty to scale applications in all namespaces. This field should have a namespace or be empty. The default value is empty.

      2

      Specifies the level of verbosity for the Custom Metrics Autoscaler Operator log messages. The allowed values are debug, info, error. The default is info.

      3

      Specifies the logging format for the Custom Metrics Autoscaler Operator log messages. The allowed values are console or json. The default is console.

      4

      Optional: Specifies one or more config maps with CA certificates, which the Custom Metrics Autoscaler Operator can use to connect securely to TLS-enabled metrics sources.

      5

      Specifies the logging level for the Custom Metrics Autoscaler Metrics Server. The allowed values are 0 for info and 4 or debug. The default is 0.

      6

      Activates audit logging for the Custom Metrics Autoscaler Operator and specifies the audit policy to use, as described in the "Configuring audit logging" section.

   5. Click Create to create the KEDA controller.

Procedure

1. Create the TriggerAuthentication or ClusterTriggerAuthentication object.

   1. Create a YAML file that defines the object:

      Example trigger authentication with a secret

      kind: TriggerAuthentication
      apiVersion: keda.sh/v1alpha1
      metadata:
        name: prom-triggerauthentication
        namespace: my-namespace
      spec:
        secretTargetRef:
        - parameter: user-name
          name: my-secret
          key: USER_NAME
        - parameter: password
          name: my-secret
          key: USER_PASSWORD

   2. Create the TriggerAuthentication object:

      $ oc create -f <filename>.yaml

2. Create or edit a ScaledObject YAML file that uses the trigger authentication:

   1. Create a YAML file that defines the object by running the following command:

      Example scaled object with a trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-triggerauthentication 1
            kind: TriggerAuthentication 2

      1

      Specify the name of your trigger authentication object.

      2

      Specify TriggerAuthentication. TriggerAuthentication is the default.

      Example scaled object with a cluster trigger authentication

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: scaledobject
        namespace: my-namespace
      spec:
        scaleTargetRef:
          name: example-deployment
        maxReplicaCount: 100
        minReplicaCount: 0
        pollingInterval: 30
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: kedatest # replace <NAMESPACE>
            metricName: http_requests_total
            threshold: '5'
            query: sum(rate(http_requests_total{job="test-app"}[1m]))
            authModes: "basic"
          authenticationRef:
            name: prom-cluster-triggerauthentication 1
            kind: ClusterTriggerAuthentication 2

      1

      Specify the name of your trigger authentication object.

      2

      Specify ClusterTriggerAuthentication.

   2. Create the scaled object by running the following command:

      $ oc apply -f <filename>

Procedure

1. Use the following command to edit the ScaledObject CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Add the autoscaling.keda.sh/paused-replicas annotation with any value:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 1
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling.

Procedure

1. Use the following command to edit the ScaledObject CR for your workload:

   $ oc edit ScaledObject scaledobject

2. Remove the autoscaling.keda.sh/paused-replicas annotation.

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "4" 1
     creationTimestamp: "2023-02-08T14:41:01Z"
     generation: 1
     name: scaledobject
     namespace: my-project
     resourceVersion: '65729'
     uid: f5aec682-acdf-4232-a783-58b5b82f5dd0

   1

   Remove this annotation to restart a paused custom metrics autoscaler.

Procedure

1. Edit the KedaController custom resource to add the auditConfig stanza:

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: keda
   spec:
   # ...
     metricsServer:
   # ...
       auditConfig:
         logFormat: "json" 1
         logOutputVolumeClaim: "pvc-audit-log" 2
         policy:
           rules: 3
           - level: Metadata
           omitStages: "RequestReceived" 4
           omitManagedFields: false 5
         lifetime: 6
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"

   1

   Specifies the output format of the audit log, either legacy or json.

   2

   Specifies an existing persistent volume claim for storing the log data. All requests coming to the API server are logged to this persistent volume claim. If you leave this field empty, the log data is sent to stdout.

   3

   Specifies which events should be recorded and what data they should include:

   • None: Do not log events.
   • Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. This is the default.
   • Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.
   • RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.

   4

   Specifies stages for which no event is created.

   5

   Specifies whether to omit the managed fields of the request and response bodies from being written to the API audit log, either true to omit the fields or false to include the fields.

   6

   Specifies the size and lifespan of the audit logs.

   • maxAge: The maximum number of days to retain audit log files, based on the timestamp encoded in their filename.
   • maxBackup: The maximum number of audit log files to retain. Set to 0 to retain all audit log files.
   • maxSize: The maximum size in megabytes of an audit log file before it gets rotated.

Verification

1. View the audit log file directly:

   1. Obtain the name of the keda-metrics-apiserver-* pod:

      oc get pod -n keda

      Example output

      NAME                                                  READY   STATUS    RESTARTS   AGE
      custom-metrics-autoscaler-operator-5cb44cd75d-9v4lv   1/1     Running   0          8m20s
      keda-metrics-apiserver-65c7cc44fd-rrl4r               1/1     Running   0          2m55s
      keda-operator-776cbb6768-zpj5b                        1/1     Running   0          2m55s

   2. View the log data by using a command similar to the following:

      $ oc logs keda-metrics-apiserver-<hash>|grep -i metadata 1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      $ oc logs keda-metrics-apiserver-65c7cc44fd-rrl4r|grep -i metadata

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4c81d41b-3dab-4675-90ce-20b87ce24013","stage":"ResponseComplete","requestURI":"/healthz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.131.0.1"],"userAgent":"kube-probe/1.28","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-02-16T13:00:03.554567Z","stageTimestamp":"2023-02-16T13:00:03.555032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
       ...

2. Alternatively, you can view a specific log:

   1. Use a command similar to the following to log into the keda-metrics-apiserver-* pod:

      $ oc rsh pod/keda-metrics-apiserver-<hash> -n keda

      For example:

      $ oc rsh pod/keda-metrics-apiserver-65c7cc44fd-rrl4r -n keda

   2. Change to the /var/audit-policy/ directory:

      sh-4.4$ cd /var/audit-policy/

   3. List the available logs:

      sh-4.4$ ls

      Example output

      log-2023.02.17-14:50  policy.yaml

   4. View the log, as needed:

      sh-4.4$ cat <log_name>/<pvc_name>|grep -i <log_level> 1

      1

      Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

      For example:

      sh-4.4$ cat log-2023.02.17-14:50/pvc-audit-log|grep -i Request

      Example output

       ...
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"63e7f68c-04ec-4f4d-8749-bf1656572a41","stage":"ResponseComplete","requestURI":"/openapi/v2","verb":"get","user":{"username":"system:aggregator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"responseStatus":{"metadata":{},"code":304},"requestReceivedTimestamp":"2023-02-17T13:12:55.035478Z","stageTimestamp":"2023-02-17T13:12:55.038346Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
       ...

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Perform one of the following:

   • To get only the Custom Metrics Autoscaler Operator must-gather data, use the following command:

     $ oc adm must-gather --image="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
     -n openshift-marketplace \
     -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     The custom image for the must-gather command is pulled directly from the Operator package manifests, so that it works on any cluster where the Custom Metric Autoscaler Operator is available.

   • To gather the default must-gather data in addition to the Custom Metric Autoscaler Operator information:

     1. Use the following command to obtain the Custom Metrics Autoscaler Operator image and set it as an environment variable:

        $ IMAGE="$(oc get packagemanifests openshift-custom-metrics-autoscaler-operator \
          -n openshift-marketplace \
          -o jsonpath='{.status.channels[?(@.name=="stable")].currentCSVDesc.annotations.containerImage}')"

     2. Use the oc adm must-gather with the Custom Metrics Autoscaler Operator image:

        $ oc adm must-gather --image-stream=openshift/must-gather --image=${IMAGE}

   Example 3.1. Example must-gather output for the Custom Metric Autoscaler

   └── keda
       ├── apps
       │   ├── daemonsets.yaml
       │   ├── deployments.yaml
       │   ├── replicasets.yaml
       │   └── statefulsets.yaml
       ├── apps.openshift.io
       │   └── deploymentconfigs.yaml
       ├── autoscaling
       │   └── horizontalpodautoscalers.yaml
       ├── batch
       │   ├── cronjobs.yaml
       │   └── jobs.yaml
       ├── build.openshift.io
       │   ├── buildconfigs.yaml
       │   └── builds.yaml
       ├── core
       │   ├── configmaps.yaml
       │   ├── endpoints.yaml
       │   ├── events.yaml
       │   ├── persistentvolumeclaims.yaml
       │   ├── pods.yaml
       │   ├── replicationcontrollers.yaml
       │   ├── secrets.yaml
       │   └── services.yaml
       ├── discovery.k8s.io
       │   └── endpointslices.yaml
       ├── image.openshift.io
       │   └── imagestreams.yaml
       ├── k8s.ovn.org
       │   ├── egressfirewalls.yaml
       │   └── egressqoses.yaml
       ├── keda.sh
       │   ├── kedacontrollers
       │   │   └── keda.yaml
       │   ├── scaledobjects
       │   │   └── example-scaledobject.yaml
       │   └── triggerauthentications
       │       └── example-triggerauthentication.yaml
       ├── monitoring.coreos.com
       │   └── servicemonitors.yaml
       ├── networking.k8s.io
       │   └── networkpolicies.yaml
       ├── keda.yaml
       ├── pods
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-ptgwx
       │   │   ├── custom-metrics-autoscaler-operator
       │   │   │   └── custom-metrics-autoscaler-operator
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── custom-metrics-autoscaler-operator-58bd9f458-ptgwx.yaml
       │   ├── custom-metrics-autoscaler-operator-58bd9f458-thbsh
       │   │   └── custom-metrics-autoscaler-operator
       │   │       └── custom-metrics-autoscaler-operator
       │   │           └── logs
       │   ├── keda-metrics-apiserver-65c7cc44fd-6wq4g
       │   │   ├── keda-metrics-apiserver
       │   │   │   └── keda-metrics-apiserver
       │   │   │       └── logs
       │   │   │           ├── current.log
       │   │   │           ├── previous.insecure.log
       │   │   │           └── previous.log
       │   │   └── keda-metrics-apiserver-65c7cc44fd-6wq4g.yaml
       │   └── keda-operator-776cbb6768-fb6m5
       │       ├── keda-operator
       │       │   └── keda-operator
       │       │       └── logs
       │       │           ├── current.log
       │       │           ├── previous.insecure.log
       │       │           └── previous.log
       │       └── keda-operator-776cbb6768-fb6m5.yaml
       ├── policy
       │   └── poddisruptionbudgets.yaml
       └── route.openshift.io
           └── routes.yaml

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Select the Administrator perspective in the Red Hat OpenShift Service on AWS web console.
2. Select Observe → Metrics.
3. To create a custom query, add your PromQL query to the Expression field.
4. To add multiple queries, select Add Query.

Procedure

1. Create a YAML file similar to the following. Only the name <2>, object name <4>, and object kind <5> are required:

   Example scaled object

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     annotations:
       autoscaling.keda.sh/paused-replicas: "0" 1
     name: scaledobject 2
     namespace: my-namespace
   spec:
     scaleTargetRef:
       apiVersion: apps/v1 3
       name: example-deployment 4
       kind: Deployment 5
       envSourceContainerName: .spec.template.spec.containers[0] 6
     cooldownPeriod:  200 7
     maxReplicaCount: 100 8
     minReplicaCount: 0 9
     metricsServer: 10
       auditConfig:
         logFormat: "json"
         logOutputVolumeClaim: "persistentVolumeClaimName"
         policy:
           rules:
           - level: Metadata
           omitStages: "RequestReceived"
           omitManagedFields: false
         lifetime:
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"
     fallback: 11
       failureThreshold: 3
       replicas: 6
     pollingInterval: 30 12
     advanced:
       restoreToOriginalReplicaCount: false 13
       horizontalPodAutoscalerConfig:
         name: keda-hpa-scale-down 14
         behavior: 15
           scaleDown:
             stabilizationWindowSeconds: 300
             policies:
             - type: Percent
               value: 100
               periodSeconds: 15
     triggers:
     - type: prometheus 16
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: kedatest
         metricName: http_requests_total
         threshold: '5'
         query: sum(rate(http_requests_total{job="test-app"}[1m]))
         authModes: basic
       authenticationRef: 17
         name: prom-triggerauthentication
         kind: TriggerAuthentication

   1

   Optional: Specifies that the Custom Metrics Autoscaler Operator is to scale the replicas to the specified value and stop autoscaling, as described in the "Pausing the custom metrics autoscaler for a workload" section.

   2

   Specifies a name for this custom metrics autoscaler.

   3

   Optional: Specifies the API version of the target resource. The default is apps/v1.

   4

   Specifies the name of the object that you want to scale.

   5

   Specifies the kind as Deployment, StatefulSet or CustomResource.

   6

   Optional: Specifies the name of the container in the target resource, from which the custom metrics autoscaler gets environment variables holding secrets and so forth. The default is .spec.template.spec.containers[0].

   7

   Optional. Specifies the period in seconds to wait after the last trigger is reported before scaling the deployment back to 0 if the minReplicaCount is set to 0. The default is 300.

   8

   Optional: Specifies the maximum number of replicas when scaling up. The default is 100.

   9

   Optional: Specifies the minimum number of replicas when scaling down.

   10

   Optional: Specifies the parameters for audit logs. as described in the "Configuring audit logging" section.

   11

   Optional: Specifies the number of replicas to fall back to if a scaler fails to get metrics from the source for the number of times defined by the failureThreshold parameter. For more information on fallback behavior, see the KEDA documentation.

   12

   Optional: Specifies the interval in seconds to check each trigger on. The default is 30.

   13

   Optional: Specifies whether to scale back the target resource to the original replica count after the scaled object is deleted. The default is false, which keeps the replica count as it is when the scaled object is deleted.

   14

   Optional: Specifies a name for the horizontal pod autoscaler. The default is keda-hpa-{scaled-object-name}.

   15

   Optional: Specifies a scaling policy to use to control the rate to scale pods up or down, as described in the "Scaling policies" section.

   16

   Specifies the trigger to use as the basis for scaling, as described in the "Understanding the custom metrics autoscaler triggers" section. This example uses Red Hat OpenShift Service on AWS monitoring.

   17

   Optional: Specifies a trigger authentication or a cluster trigger authentication. For more information, see Understanding the custom metrics autoscaler trigger authentication in the Additional resources section.

   • Enter TriggerAuthentication to use a trigger authentication. This is the default.
   • Enter ClusterTriggerAuthentication to use a cluster trigger authentication.

2. Create the custom metrics autoscaler by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. In the Red Hat OpenShift Service on AWS web console, click Operators → Installed Operators.
2. Switch to the keda project.

3. Remove the KedaController custom resource.

   1. Find the CustomMetricsAutoscaler Operator and click the KedaController tab.
   2. Find the custom resource, and then click Delete KedaController.
   3. Click Uninstall.

4. Remove the Custom Metrics Autoscaler Operator:

   1. Click Operators → Installed Operators.
   2. Find the CustomMetricsAutoscaler Operator and click the Options menu and select Uninstall Operator.
   3. Click Uninstall.

5. Optional: Use the OpenShift CLI to remove the custom metrics autoscaler components:

   1. Delete the custom metrics autoscaler CRDs:

      • clustertriggerauthentications.keda.sh
      • kedacontrollers.keda.sh
      • scaledjobs.keda.sh
      • scaledobjects.keda.sh
      • triggerauthentications.keda.sh

      $ oc delete crd clustertriggerauthentications.keda.sh kedacontrollers.keda.sh scaledjobs.keda.sh scaledobjects.keda.sh triggerauthentications.keda.sh

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings. However, there might be a few cluster roles that must be manually deleted.

   2. List any custom metrics autoscaler cluster roles:

      $ oc get clusterrole | grep keda.sh

   3. Delete the listed custom metrics autoscaler cluster roles. For example:

      $ oc delete clusterrole.keda.sh-v1alpha1-admin

   4. List any custom metrics autoscaler cluster role bindings:

      $ oc get clusterrolebinding | grep keda.sh

   5. Delete the listed custom metrics autoscaler cluster role bindings. For example:

      $ oc delete clusterrolebinding.keda.sh-v1alpha1-admin

6. Delete the custom metrics autoscaler project:

   $ oc delete project keda

7. Delete the Cluster Metric Autoscaler Operator:

   $ oc delete operator/openshift-custom-metrics-autoscaler-operator.keda

Procedure

1. Create a pod with a specific label in the pod spec:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1
        labels:
          security: S1
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: security-s1
          image: docker.io/ocpqe/hello-pod
          securityContext:
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

2. When creating other pods, configure the following parameters to add the affinity:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1-east
      # ...
      spec:
        affinity: 1
          podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution: 2
            - labelSelector:
                matchExpressions:
                - key: security 3
                  values:
                  - S1
                  operator: In 4
              topologyKey: topology.kubernetes.io/zone 5
      # ...

      1

      Adds a pod affinity.

      2

      Configures the requiredDuringSchedulingIgnoredDuringExecution parameter or the preferredDuringSchedulingIgnoredDuringExecution parameter.

      3

      Specifies the key and values that must be met. If you want the new pod to be scheduled with the other pod, use the same key and values parameters as the label on the first pod.

      4

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

      5

      Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

Procedure

1. Create a pod with a specific label in the pod spec:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s1
        labels:
          security: S1
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: security-s1
          image: docker.io/ocpqe/hello-pod
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

2. When creating other pods, configure the following parameters:

   1. Create a YAML file with the following content:

      apiVersion: v1
      kind: Pod
      metadata:
        name: security-s2-east
      # ...
      spec:
      # ...
        affinity: 1
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 2
            - weight: 100 3
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: security 4
                    values:
                    - S1
                    operator: In 5
                topologyKey: kubernetes.io/hostname 6
      # ...

      1

      Adds a pod anti-affinity.

      2

      Configures the requiredDuringSchedulingIgnoredDuringExecution parameter or the preferredDuringSchedulingIgnoredDuringExecution parameter.

      3

      For a preferred rule, specifies a weight for the node, 1-100. The node that with highest weight is preferred.

      4

      Specifies the key and values that must be met. If you want the new pod to not be scheduled with the other pod, use the same key and values parameters as the label on the first pod.

      5

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

      6

      Specifies a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

   2. Create the pod.

      $ oc create -f <pod-spec>.yaml

1. Create a pod with a specific label in the pod spec:

   1. Create a YAML file with the following content:

      Note

      You cannot add an affinity directly to a scheduled pod.

      Example output

      apiVersion: v1
      kind: Pod
      metadata:
        name: s1
      spec:
        affinity: 1
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution: 2
              nodeSelectorTerms:
              - matchExpressions:
                - key: e2e-az-name 3
                  values:
                  - e2e-az1
                  - e2e-az2
                  operator: In 4
      #...

      1

      Adds a pod affinity.

      2

      Configures the requiredDuringSchedulingIgnoredDuringExecution parameter.

      3

      Specifies the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and values parameters as the label in the node.

      4

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

   2. Create the pod:

      $ oc create -f <file-name>.yaml

1. Create a pod with a specific label:

   1. Create a YAML file with the following content:

      Note

      You cannot add an affinity directly to a scheduled pod.

      apiVersion: v1
      kind: Pod
      metadata:
        name: s1
      spec:
        affinity: 1
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 2
            - weight: 3
              preference:
                matchExpressions:
                - key: e2e-az-name 4
                  values:
                  - e2e-az3
                  operator: In 5
      #...

      1

      Adds a pod affinity.

      2

      Configures the preferredDuringSchedulingIgnoredDuringExecution parameter.

      3

      Specifies a weight for the node, as a number 1-100. The node with highest weight is preferred.

      4

      Specifies the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and values parameters as the label in the node.

      5

      Specifies an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.

   2. Create the pod.

      $ oc create -f <file-name>.yaml

1. Define the daemon set yaml file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset 1
     template:
       metadata:
         labels:
           name: hello-daemonset 2
       spec:
         nodeSelector: 3
           role: worker
         containers:
         - image: openshift/hello-openshift
           imagePullPolicy: Always
           name: registry
           ports:
           - containerPort: 80
             protocol: TCP
           resources: {}
           terminationMessagePath: /dev/termination-log
         serviceAccount: default
         terminationGracePeriodSeconds: 10
   #...

   1

   The label selector that determines which pods belong to the daemon set.

   2

   The pod template’s label selector. Must match the label selector above.

   3

   The node selector that determines on which nodes pod replicas should be deployed. A matching label must be present on the node.

2. Create the daemon set object:

   $ oc create -f daemonset.yaml

3. To verify that the pods were created, and that each node has a pod replica:

   1. Find the daemonset pods:

      $ oc get pods

      Example output

      hello-daemonset-cx6md   1/1       Running   0          2m
      hello-daemonset-e3md9   1/1       Running   0          2m

   2. View the pods to verify the pod has been placed onto the node:

      $ oc describe pod/hello-daemonset-cx6md|grep Node

      Example output

      Node:        openshift-node01.hostname.com/10.14.20.134

      $ oc describe pod/hello-daemonset-e3md9|grep Node

      Example output

      Node:        openshift-node02.hostname.com/10.14.20.137

1. Create a YAML file similar to the following:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: pi
   spec:
     parallelism: 1    1
     completions: 1    2
     activeDeadlineSeconds: 1800 3
     backoffLimit: 6   4
     template:         5
       metadata:
         name: pi
       spec:
         containers:
         - name: pi
           image: perl
           command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
         restartPolicy: OnFailure    6
   #...

   1

   Optional: Specify how many pod replicas a job should run in parallel; defaults to 1.

   • For non-parallel jobs, leave unset. When unset, defaults to 1.

   2

   Optional: Specify how many successful pod completions are needed to mark a job completed.

   • For non-parallel jobs, leave unset. When unset, defaults to 1.
   • For parallel jobs with a fixed completion count, specify the number of completions.
   • For parallel jobs with a work queue, leave unset. When unset defaults to the parallelism value.

   3

   Optional: Specify the maximum duration the job can run.

   4

   Optional: Specify the number of retries for a job. This field defaults to six.

   5

   Specify the template for the pod the controller creates.

   6

   Specify the restart policy of the pod:

   • Never. Do not restart the job.
   • OnFailure. Restart the job only if it fails.

   • Always. Always restart the job.

     For details on how Red Hat OpenShift Service on AWS uses restart policy with failed containers, see the Example States in the Kubernetes documentation.

2. Create the job:

   $ oc create -f <file-name>.yaml

1. Create a YAML file similar to the following:

   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: pi
   spec:
     schedule: "*/1 * * * *"          1
     concurrencyPolicy: "Replace"     2
     startingDeadlineSeconds: 200     3
     suspend: true                    4
     successfulJobsHistoryLimit: 3    5
     failedJobsHistoryLimit: 1        6
     jobTemplate:                     7
       spec:
         template:
           metadata:
             labels:                  8
               parent: "cronjobpi"
           spec:
             containers:
             - name: pi
               image: perl
               command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
             restartPolicy: OnFailure 9

   1

   Schedule for the job specified in cron format. In this example, the job will run every minute.

   2

   An optional concurrency policy, specifying how to treat concurrent jobs within a cron job. Only one of the following concurrent policies may be specified. If not specified, this defaults to allowing concurrent executions.

   • Allow allows cron jobs to run concurrently.
   • Forbid forbids concurrent runs, skipping the next run if the previous has not finished yet.
   • Replace cancels the currently running job and replaces it with a new one.

   3

   An optional deadline (in seconds) for starting the job if it misses its scheduled time for any reason. Missed jobs executions will be counted as failed ones. If not specified, there is no deadline.

   4

   An optional flag allowing the suspension of a cron job. If set to true, all subsequent executions will be suspended.

   5

   The number of successful finished jobs to retain (defaults to 3).

   6

   The number of failed finished jobs to retain (defaults to 1).

   7

   Job template. This is similar to the job example.

   8

   Sets a label for jobs spawned by this cron job.

   9

   The restart policy of the pod. This does not apply to the job controller.

   Note

   The .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 3 and 1 respectively. Setting a limit to 0 corresponds to keeping none of the corresponding kind of jobs after they finish.

2. Create the cron job:

   $ oc create -f <file-name>.yaml

Procedure

1. Mark the nodes unschedulable before performing the pod evacuation.

   1. Mark the node as unschedulable:

      $ oc adm cordon <node1>

      Example output

      node/<node1> cordoned

   2. Check that the node status is Ready,SchedulingDisabled:

      $ oc get node <node1>

      Example output

      NAME        STATUS                     ROLES     AGE       VERSION
      <node1>     Ready,SchedulingDisabled   worker    1d        v1.30.3

2. Evacuate the pods using one of the following methods:

   • Evacuate all or selected pods on one or more nodes:

     $ oc adm drain <node1> <node2> [--pod-selector=<pod_selector>]

   • Force the deletion of bare pods using the --force option. When set to true, deletion continues even if there are pods not managed by a replication controller, replica set, job, daemon set, or stateful set:

     $ oc adm drain <node1> <node2> --force=true

   • Set a period of time in seconds for each pod to terminate gracefully, use --grace-period. If negative, the default value specified in the pod will be used:

     $ oc adm drain <node1> <node2> --grace-period=-1

   • Ignore pods managed by daemon sets using the --ignore-daemonsets flag set to true:

     $ oc adm drain <node1> <node2> --ignore-daemonsets=true

   • Set the length of time to wait before giving up using the --timeout flag. A value of 0 sets an infinite length of time:

     $ oc adm drain <node1> <node2> --timeout=5s

   • Delete pods even if there are pods using emptyDir volumes by setting the --delete-emptydir-data flag to true. Local data is deleted when the node is drained:

     $ oc adm drain <node1> <node2> --delete-emptydir-data=true

   • List objects that will be migrated without actually performing the evacuation, using the --dry-run option set to true:

     $ oc adm drain <node1> <node2>  --dry-run=true

     Instead of specifying specific node names (for example, <node1> <node2>), you can use the --selector=<node_selector> option to evacuate pods on selected nodes.

3. Mark the node as schedulable when done.

   $ oc adm uncordon <node1>

Procedure

1. Create the pod for the Init Container:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: myapp-pod
        labels:
          app: myapp
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: myapp-container
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'echo The app is running! && sleep 3600']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
        initContainers:
        - name: init-myservice
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'until getent hosts myservice; do echo waiting for myservice; sleep 2; done;']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
        - name: init-mydb
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'until getent hosts mydb; do echo waiting for mydb; sleep 2; done;']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

   2. Create the pod:

      $ oc create -f myapp.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     0/1       Init:0/2            0          5s

      The pod status, Init:0/2, indicates it is waiting for the two services.

2. Create the myservice service.

   1. Create a YAML file similar to the following:

      kind: Service
      apiVersion: v1
      metadata:
        name: myservice
      spec:
        ports:
        - protocol: TCP
          port: 80
          targetPort: 9376

   2. Create the pod:

      $ oc create -f myservice.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     0/1       Init:1/2            0          5s

      The pod status, Init:1/2, indicates it is waiting for one service, in this case the mydb service.

3. Create the mydb service:

   1. Create a YAML file similar to the following:

      kind: Service
      apiVersion: v1
      metadata:
        name: mydb
      spec:
        ports:
        - protocol: TCP
          port: 80
          targetPort: 9377

   2. Create the pod:

      $ oc create -f mydb.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     1/1       Running             0          2m

      The pod status indicated that it is no longer waiting for the services and is running.