Chapter 2. Working with pods

1. Change to the project:

   $ oc project <project-name>

2. Run the following command:

   $ oc get pods

   For example:

   $ oc get pods

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   console-698d866b78-bnshf   1/1     Running   2          165m
   console-698d866b78-m87pm   1/1     Running   2          165m

   Add the

   -o wide

   flags to view the pod IP address and the node where the pod is located.

   $ oc get pods -o wide

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                           NOMINATED NODE
   console-698d866b78-bnshf   1/1     Running   2          166m   10.128.0.24   ip-10-0-152-71.ec2.internal    <none>
   console-698d866b78-m87pm   1/1     Running   2          166m   10.129.0.23   ip-10-0-173-237.ec2.internal   <none>

1. Run the following command:

   $ oc adm top pods

   For example:

   $ oc adm top pods -n openshift-console

   Example output

   NAME                         CPU(cores)   MEMORY(bytes)
   console-7f58c69899-q8c8k     0m           22Mi
   console-7f58c69899-xhbgg     0m           25Mi
   downloads-594fcccf94-bcxk8   3m           18Mi
   downloads-594fcccf94-kv4p6   2m           15Mi

2. Run the following command to view the usage statistics for pods with labels:

   $ oc adm top pod --selector=''

   You must choose the selector (label query) to filter on. Supports

   =

   ,

   ==

   , and

   !=

   .

   For example:

   $ oc adm top pod --selector='name=my-pod'

Procedure (UI)

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.

   Note

   Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.

2. Select a project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs.

1. Write an object definition JSON file, and specify the data traffic speed using

   kubernetes.io/ingress-bandwidth

   and

   kubernetes.io/egress-bandwidth

   annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod object definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

1. Create a

   Pod

   spec or edit existing pods to include the

   system-cluster-critical

   priority class:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pdb
   spec:
     template:
       metadata:
         name: critical-pod
       priorityClassName: system-cluster-critical 

   1

   1

   Default priority class for pods that should never be evicted from a node.

   Alternatively, you can specify

   system-node-critical

   for pods that are important to the cluster but can be removed if necessary.

2. Create the pod:

   $ oc create -f <file-name>.yaml

1. In the Topology view, click the node to reveal the side pane.

2. From the Actions drop-down list, select Add HorizontalPodAutoscaler to open the Add HorizontalPodAutoscaler form.

   Figure 2.2. Add HorizontalPodAutoscaler

   

3. From the Add HorizontalPodAutoscaler form, define the name, minimum and maximum pod limits, the CPU and memory usage, and click Save.

   Note

   If any of the values for CPU and memory usage are missing, a warning is displayed.

1. In the Topology view, click the node to reveal the side pane.
2. From the Actions drop-down list, select Edit HorizontalPodAutoscaler to open the Edit Horizontal Pod Autoscaler form.
3. From the Edit Horizontal Pod Autoscaler form, edit the minimum and maximum pod limits and the CPU and memory usage, and click Save.

1. In the Topology view, click the node to reveal the side panel.
2. From the Actions drop-down list, select Remove HorizontalPodAutoscaler.
3. In the confirmation pop-up window, click Remove to remove the HPA.

1. Perform one of the following:

   • To scale based on the percent of CPU utilization, create a

     HorizontalPodAutoscaler

     object for an existing object:

     $ oc autoscale <object_type>/<name> \

     1

     
       --min <number> \

     2

     
       --max <number> \

     3

     
       --cpu-percent=<percent> 

     4

     1

     Specify the type and name of the object to autoscale. The object must exist and be a Deployment, DeploymentConfig/dc, ReplicaSet/rs, ReplicationController/rc, or StatefulSet.

     2

     Optionally, specify the minimum number of replicas when scaling down.

     3

     Specify the maximum number of replicas when scaling up.

     4

     Specify the target average CPU utilization over all the pods, represented as a percent of requested CPU. If not specified or negative, a default autoscaling policy is used.

     For example, the following command shows autoscaling for the

     image-registry

     Deployment

     object. The initial deployment requires 3 pods. The HPA object increases the minimum to 5. If CPU usage on the pods reaches 75%, the pods will increase to 7:

     $ oc autoscale deployment/image-registry --min=5 --max=7 --cpu-percent=75

   • To scale for a specific CPU value, create a YAML file similar to the following for an existing object:

     1. Create a YAML file similar to the following:

        apiVersion: autoscaling/v2 

        1

        
        kind: HorizontalPodAutoscaler
        metadata:
          name: cpu-autoscale 

        2

        
          namespace: default
        spec:
          scaleTargetRef:
            apiVersion: apps/v1 

        3

        
            kind: Deployment 

        4

        
            name: example 

        5

        
          minReplicas: 1 

        6

        
          maxReplicas: 10 

        7

        
          metrics: 

        8

        
          - type: Resource
            resource:
              name: cpu 

        9

        
              target:
                type: AverageValue 

        10

        
                averageValue: 500m 

        11

        1

        Use the autoscaling/v2 API.

        2

        Specify a name for this horizontal pod autoscaler object.

        3

        Specify the API version of the object to scale:

        • For a

          Deployment

          ,

          ReplicaSet

          ,

          Statefulset

          object, use

          apps/v1

          .
        • For a

          ReplicationController

          , use

          v1

          .
        • For a

          DeploymentConfig

          , use

          apps.openshift.io/v1

          .

        4

        Specify the type of object. The object must be a Deployment, DeploymentConfig/dc, ReplicaSet/rs, ReplicationController/rc, or StatefulSet.

        5

        Specify the name of the object to scale. The object must exist.

        6

        Specify the minimum number of replicas when scaling down.

        7

        Specify the maximum number of replicas when scaling up.

        8

        Use the metrics parameter for memory utilization.

        9

        Specify cpu for CPU utilization.

        10

        Set to AverageValue.

        11

        Set to averageValue with the targeted CPU value.

     2. Create the horizontal pod autoscaler:

        $ oc create -f <file-name>.yaml

2. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa cpu-autoscale

   Example output

   NAME            REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   cpu-autoscale   Deployment/example   173m/500m       1         10        1          20m

1. Create a YAML file for one of the following:

   • To scale for a specific memory value, create a

     HorizontalPodAutoscaler

     object similar to the following for an existing object:

     apiVersion: autoscaling/v2 

     1

     
     kind: HorizontalPodAutoscaler
     metadata:
       name: hpa-resource-metrics-memory 

     2

     
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: apps/v1 

     3

     
         kind: Deployment 

     4

     
         name: example 

     5

     
       minReplicas: 1 

     6

     
       maxReplicas: 10 

     7

     
       metrics: 

     8

     
       - type: Resource
         resource:
           name: memory 

     9

     
           target:
             type: AverageValue 

     10

     
             averageValue: 500Mi 

     11

     
       behavior: 

     12

     
         scaleDown:
           stabilizationWindowSeconds: 300
           policies:
           - type: Pods
             value: 4
             periodSeconds: 60
           - type: Percent
             value: 10
             periodSeconds: 60
           selectPolicy: Max

     1

     Use the autoscaling/v2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a

       Deployment

       ,

       ReplicaSet

       , or

       Statefulset

       object, use

       apps/v1

       .
     • For a

       ReplicationController

       , use

       v1

       .
     • For a

       DeploymentConfig

       , use

       apps.openshift.io/v1

       .

     4

     Specify the type of object. The object must be a Deployment, DeploymentConfig, ReplicaSet, ReplicationController, or StatefulSet.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set the type to AverageValue.

     11

     Specify averageValue and a specific memory value.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

   • To scale for a percentage, create a

     HorizontalPodAutoscaler

     object similar to the following for an existing object:

     apiVersion: autoscaling/v2 

     1

     
     kind: HorizontalPodAutoscaler
     metadata:
       name: memory-autoscale 

     2

     
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: apps/v1 

     3

     
         kind: Deployment 

     4

     
         name: example 

     5

     
       minReplicas: 1 

     6

     
       maxReplicas: 10 

     7

     
       metrics: 

     8

     
       - type: Resource
         resource:
           name: memory 

     9

     
           target:
             type: Utilization 

     10

     
             averageUtilization: 50 

     11

     
       behavior: 

     12

     
         scaleUp:
           stabilizationWindowSeconds: 180
           policies:
           - type: Pods
             value: 6
             periodSeconds: 120
           - type: Percent
             value: 10
             periodSeconds: 120
           selectPolicy: Max

     1

     Use the autoscaling/v2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a ReplicationController, use

       v1

       .
     • For a DeploymentConfig, use

       apps.openshift.io/v1

       .
     • For a Deployment, ReplicaSet, Statefulset object, use

       apps/v1

       .

     4

     Specify the type of object. The object must be a Deployment, DeploymentConfig, ReplicaSet, ReplicationController, or StatefulSet.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set to Utilization.

     11

     Specify averageUtilization and a target average memory utilization over all the pods, represented as a percent of requested memory. The target pods must have memory requests configured.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

2. Create the horizontal pod autoscaler:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f hpa.yaml

   Example output

   horizontalpodautoscaler.autoscaling/hpa-resource-metrics-memory created

3. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa hpa-resource-metrics-memory

   Example output

   NAME                          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   hpa-resource-metrics-memory   Deployment/example   2441216/500Mi   1         10        1          20m

   $ oc describe hpa hpa-resource-metrics-memory

   Example output

   Name:                        hpa-resource-metrics-memory
   Namespace:                   default
   Labels:                      <none>
   Annotations:                 <none>
   CreationTimestamp:           Wed, 04 Mar 2020 16:31:37 +0530
   Reference:                   Deployment/example
   Metrics:                     ( current / target )
     resource memory on pods:   2441216 / 500Mi
   Min replicas:                1
   Max replicas:                10
   ReplicationController pods:  1 current / 1 desired
   Conditions:
     Type            Status  Reason              Message
     ----            ------  ------              -------
     AbleToScale     True    ReadyForNewScale    recommended size matches current size
     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
   Events:
     Type     Reason                   Age                 From                       Message
     ----     ------                   ----                ----                       -------
     Normal   SuccessfulRescale        6m34s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose VerticalPodAutoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory

   openshift-vertical-pod-autoscaler

   namespace, which is automatically created if it does not exist.
4. Click Install.

5. Verify the installation by listing the VPA Operator components:

   1. Navigate to Workloads Pods.
   2. Select the

      openshift-vertical-pod-autoscaler

      project from the drop-down menu and verify that there are four pods running.
   3. Navigate to Workloads Deployments to verify that there are four deployments running.

6. Optional. Verify the installation in the OpenShift Container Platform CLI using the following command:

   $ oc get all -n openshift-vertical-pod-autoscaler

   The output shows four pods and four deplyoments:

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/vertical-pod-autoscaler-operator-85b4569c47-2gmhc   1/1     Running   0          3m13s
   pod/vpa-admission-plugin-default-67644fc87f-xq7k9       1/1     Running   0          2m56s
   pod/vpa-recommender-default-7c54764b59-8gckt            1/1     Running   0          2m56s
   pod/vpa-updater-default-7f6cc87858-47vw9                1/1     Running   0          2m56s
   
   NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   service/vpa-webhook   ClusterIP   172.30.53.206   <none>        443/TCP   2m56s
   
   NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/vertical-pod-autoscaler-operator   1/1     1            1           3m13s
   deployment.apps/vpa-admission-plugin-default       1/1     1            1           2m56s
   deployment.apps/vpa-recommender-default            1/1     1            1           2m56s
   deployment.apps/vpa-updater-default                1/1     1            1           2m56s
   
   NAME                                                          DESIRED   CURRENT   READY   AGE
   replicaset.apps/vertical-pod-autoscaler-operator-85b4569c47   1         1         1       3m13s
   replicaset.apps/vpa-admission-plugin-default-67644fc87f       1         1         1       2m56s
   replicaset.apps/vpa-recommender-default-7c54764b59            1         1         1       2m56s
   replicaset.apps/vpa-updater-default-7f6cc87858                1         1         1       2m56s

1. Change to the project where the workload object you want to scale is located.

   1. Create a VPA CR YAML file:

      apiVersion: autoscaling.k8s.io/v1
      kind: VerticalPodAutoscaler
      metadata:
        name: vpa-recommender
      spec:
        targetRef:
          apiVersion: "apps/v1"
          kind:       Deployment 

      1

      
          name:       frontend 

      2

      
        updatePolicy:
          updateMode: "Auto" 

      3

      
        resourcePolicy: 

      4

      
          containerPolicies:
          - containerName: my-opt-sidecar
            mode: "Off"
        recommenders: 

      5

      
          - name: my-recommender

      1

      Specify the type of workload object you want this VPA to manage: Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, or ReplicationController.

      2

      Specify the name of an existing workload object you want this VPA to manage.

      3

      Specify the VPA mode:

      • auto

        to automatically apply the recommended resources on pods associated with the controller. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests.

      • recreate

        to automatically apply the recommended resources on pods associated with the workload object. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests. The

        recreate

        mode should be used rarely, only if you need to ensure that the pods are restarted whenever the resource request changes.

      • initial

        to automatically apply the recommended resources when pods associated with the workload object are created. The VPA does not update the pods as it learns new resource recommendations.

      • off

        to only generate resource recommendations for the pods associated with the workload object. The VPA does not update the pods as it learns new resource recommendations and does not apply the recommendations to new pods.

      4

      Optional. Specify the containers you want to opt-out and set the mode to Off.

      5

      Optional. Specify an alternative recommender.

   2. Create the VPA CR:

      $ oc create -f <file-name>.yaml

      After a few moments, the VPA learns the resource usage of the containers in the pods associated with the workload object.

      You can view the VPA recommendations using the following command:

      $ oc get vpa <vpa-name> --output yaml

      The output shows the recommendations for CPU and memory requests, similar to the following:

      Example output

      ...
      status:
      
      ...
      
        recommendation:
          containerRecommendations:
          - containerName: frontend
            lowerBound: 

      1

      
              cpu: 25m
              memory: 262144k
            target: 

      2

      
              cpu: 25m
              memory: 262144k
            uncappedTarget: 

      3

      
              cpu: 25m
              memory: 262144k
            upperBound: 

      4

      
              cpu: 262m
              memory: "274357142"
          - containerName: backend
            lowerBound:
              cpu: 12m
              memory: 131072k
            target:
              cpu: 12m
              memory: 131072k
            uncappedTarget:
              cpu: 12m
              memory: 131072k
            upperBound:
              cpu: 476m
              memory: "498558823"
      
      ...

      1

      lowerBound is the minimum recommended resource levels.

      2

      target is the recommended resource levels.

      3

      upperBound is the highest recommended resource levels.

      4

      uncappedTarget is the most recent resource recommendations.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. Switch to the openshift-vertical-pod-autoscaler project.
3. For the VerticalPodAutoscaler Operator, click the Options menu and select Uninstall Operator.
4. Optional: To remove all operands associated with the Operator, in the dialog box, select Delete all operand instances for this operator checkbox.
5. Click Uninstall.

6. Optional: Use the OpenShift CLI to remove the VPA components:

   1. Delete the VPA namespace:

      $ oc delete namespace openshift-vertical-pod-autoscaler

   2. Delete the VPA custom resource definition (CRD) objects:

      $ oc delete crd verticalpodautoscalercheckpoints.autoscaling.k8s.io

      $ oc delete crd verticalpodautoscalercontrollers.autoscaling.openshift.io

      $ oc delete crd verticalpodautoscalers.autoscaling.k8s.io

      Deleting the CRDs removes the associated roles, cluster roles, and role bindings.

      Note

      This action removes from the cluster all user-created VPA CRs. If you re-install the VPA, you must create these objects again.

   3. Delete the VPA Operator:

      $ oc delete operator/vertical-pod-autoscaler.openshift-vertical-pod-autoscaler

1. Create a secret object that contains the data you want to keep secret. The specific data required for each secret type is descibed in the following sections.

   Example YAML object that creates an opaque secret

   apiVersion: v1
   kind: Secret
   metadata:
     name: test-secret
   type: Opaque 

   1

   
   data: 

   2

   
     username: <username>
     password: <password>
   stringData: 

   3

   
     hostname: myapp.mydomain.com
     secret.properties: |
       property1=valueA
       property2=valueB

   1

   Specifies the type of secret.

   2

   Specifies encoded string and data.

   3

   Specifies decoded string and data.

   Use either the

   data

   or

   stringdata

   fields, not both.

2. Update the pod’s service account to reference the secret:

   YAML of a service account that uses a secret

   apiVersion: v1
   kind: ServiceAccount
    ...
   secrets:
   - name: test-secret

3. Create a pod, which consumes the secret as an environment variable or as a file (using a

   secret

   volume):

   YAML of a pod populating files in a volume with secret data

   apiVersion: v1
   kind: Pod
   metadata:
     name: secret-example-pod
   spec:
     containers:
       - name: secret-test-container
         image: busybox
         command: [ "/bin/sh", "-c", "cat /etc/secret-volume/*" ]
         volumeMounts: 

   1

   
             - name: secret-volume
               mountPath: /etc/secret-volume 

   2

   
               readOnly: true 

   3

   
     volumes:
       - name: secret-volume
         secret:
           secretName: test-secret 

   4

   
     restartPolicy: Never

   1

   Add a volumeMounts field to each container that needs the secret.

   2

   Specifies an unused directory name where you would like the secret to appear. Each key in the secret data map becomes the filename under mountPath.

   3

   Set to true. If true, this instructs the driver to provide a read-only volume.

   4

   Specifies the name of the secret.

   YAML of a pod populating environment variables with secret data

   apiVersion: v1
   kind: Pod
   metadata:
     name: secret-example-pod
   spec:
     containers:
       - name: secret-test-container
         image: busybox
         command: [ "/bin/sh", "-c", "export" ]
         env:
           - name: TEST_SECRET_USERNAME_ENV_VAR
             valueFrom:
               secretKeyRef: 

   1

   
                 name: test-secret
                 key: username
     restartPolicy: Never

   1

   Specifies the environment variable that consumes the secret key.

   YAML of a build config populating environment variables with secret data

   apiVersion: build.openshift.io/v1
   kind: BuildConfig
   metadata:
     name: secret-example-bc
   spec:
     strategy:
       sourceStrategy:
         env:
         - name: TEST_SECRET_USERNAME_ENV_VAR
           valueFrom:
             secretKeyRef: 

   1

   
               name: test-secret
               key: username
         from:
           kind: ImageStreamTag
           namespace: openshift
           name: 'cli:latest'

   1

   Specifies the environment variable that consumes the secret key.

Procedure

1. Create a service account in your namespace by running the following command:

   $ oc create sa <service_account_name> -n <your_namespace>

2. Save the following YAML example to a file named

   service-account-token-secret.yaml

   . The example includes a

   Secret

   object configuration that you can use to generate a service account token:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name> 

   1

   
     annotations:
       kubernetes.io/service-account.name: "sa-name" 

   2

   
   type: kubernetes.io/service-account-token 

   3

   1

   Replace <secret_name> with the name of your service token secret.

   2

   Specifies an existing service account name. If you are creating both the ServiceAccount and the Secret objects, create the ServiceAccount object first.

   3

   Specifies a service account token secret type.

3. Generate the service account token by applying the file:

   $ oc apply -f service-account-token-secret.yaml

4. Get the service account token from the secret by running the following command:

   $ oc get secret <sa_token_secret> -o jsonpath='{.data.token}' | base64 --decode 

   1

   Example output

   ayJhbGciOiJSUzI1NiIsImtpZCI6IklOb2dtck1qZ3hCSWpoNnh5YnZhSE9QMkk3YnRZMVZoclFfQTZfRFp1YlUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImJ1aWxkZXItdG9rZW4tdHZrbnIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYnVpbGRlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNmZGU2MGZmLTA1NGYtNDkyZi04YzhjLTNlZjE0NDk3MmFmNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmJ1aWxkZXIifQ.OmqFTDuMHC_lYvvEUrjr1x453hlEEHYcxS9VKSzmRkP1SiVZWPNPkTWlfNRp6bIUZD3U6aN3N7dMSN0eI5hu36xPgpKTdvuckKLTCnelMx6cxOdAbrcw1mCmOClNscwjS1KO1kzMtYnnq8rXHiMJELsNlhnRyyIXRTtNBsy4t64T3283s3SLsancyx0gy0ujx-Ch3uKAKdZi5iT-I8jnnQ-ds5THDs2h65RJhgglQEmSxpHrLGZFmyHAQI-_SjvmHZPXEc482x3SkaQHNLqpmrpJorNqh1M8ZHKzlujhZgVooMvJmWPXTb2vnvi3DGn2XI-hZxl1yD2yGH1RBpYUHA

   1

   Replace <sa_token_secret> with the name of your service token secret.

5. Use your service account token to authenticate with the API of your cluster:

   $ curl -X GET <openshift_cluster_api> --header "Authorization: Bearer <token>" 

   1

   2

   1

   Replace <openshift_cluster_api> with the OpenShift cluster API.

   2

   Replace <token> with the service account token that is output in the preceding command.

1. Delete the secret:

   $ oc delete secret <secret_name>

2. Clear the annotations:

   $ oc annotate service <service_name> service.beta.openshift.io/serving-cert-generation-error-

   $ oc annotate service <service_name> service.beta.openshift.io/serving-cert-generation-error-num-