Chapter 2. Working with pods

1. Create a YAML file with the an object definition similar to the following:

   apiVersion: policy/v1 1
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     minAvailable: 2  2
     selector:  3
       matchLabels:
         name: my-pod

   1

   PodDisruptionBudget is part of the policy/v1 API group.

   2

   The minimum number of pods that must be available simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.

   3

   A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined. Leave this parameter blank, for example selector {}, to select all pods in the project.

   Or:

   apiVersion: policy/v1 1
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     maxUnavailable: 25% 2
     selector: 3
       matchLabels:
         name: my-pod

   1

   PodDisruptionBudget is part of the policy/v1 API group.

   2

   The maximum number of pods that can be unavailable simultaneously. This can be either an integer or a string specifying a percentage, for example, 20%.

   3

   A label query over a set of resources. The result of matchLabels and matchExpressions are logically conjoined. Leave this parameter blank, for example selector {}, to select all pods in the project.

2. Run the following command to add the object to project:

   $ oc create -f </path/to/file> -n <project_name>

Procedure

1. Create a YAML file that defines a PodDisruptionBudget object and specify the unhealthy pod eviction policy:

   Example pod-disruption-budget.yaml file

   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: my-pdb
   spec:
     minAvailable: 2
     selector:
       matchLabels:
         name: my-pod
     unhealthyPodEvictionPolicy: AlwaysAllow 1

   1

   Choose either IfHealthyBudget or AlwaysAllow as the unhealthy pod eviction policy. The default is IfHealthyBudget when the unhealthyPodEvictionPolicy field is empty.

2. Create the PodDisruptionBudget object by running the following command:

   $ oc create -f pod-disruption-budget.yaml

1. View the new state of the deployment:

   $ oc get deployment image-registry

   There are now 5 pods in the deployment:

   Example output

   NAME             REVISION   DESIRED   CURRENT   TRIGGERED BY
   image-registry   1          5         5         config

1. Create a service account for the alternative recommender and bind that service account to the required cluster role:

   apiVersion: v1 1
   kind: ServiceAccount
   metadata:
     name: alt-vpa-recommender-sa
     namespace: <namespace_name>
   ---
   apiVersion: rbac.authorization.k8s.io/v1 2
   kind: ClusterRoleBinding
   metadata:
     name: system:example-metrics-reader
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:metrics-reader
   subjects:
   - kind: ServiceAccount
     name: alt-vpa-recommender-sa
     namespace: <namespace_name>
   ---
   apiVersion: rbac.authorization.k8s.io/v1 3
   kind: ClusterRoleBinding
   metadata:
     name: system:example-vpa-actor
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:vpa-actor
   subjects:
   - kind: ServiceAccount
     name: alt-vpa-recommender-sa
     namespace: <namespace_name>
   ---
   apiVersion: rbac.authorization.k8s.io/v1 4
   kind: ClusterRoleBinding
   metadata:
     name: system:example-vpa-target-reader-binding
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:vpa-target-reader
   subjects:
   - kind: ServiceAccount
     name: alt-vpa-recommender-sa
     namespace: <namespace_name>

   1

   Creates a service account for the recommender in the namespace where the recommender is deployed.

   2

   Binds the recommender service account to the metrics-reader role. Specify the namespace where the recommender is to be deployed.

   3

   Binds the recommender service account to the vpa-actor role. Specify the namespace where the recommender is to be deployed.

   4

   Binds the recommender service account to the vpa-target-reader role. Specify the namespace where the recommender is to be deployed.

2. To add the alternative recommender to the cluster, create a Deployment object similar to the following:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: alt-vpa-recommender
     namespace: <namespace_name>
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: alt-vpa-recommender
     template:
       metadata:
         labels:
           app: alt-vpa-recommender
       spec:
         containers: 1
         - name: recommender
           image: quay.io/example/alt-recommender:latest 2
           imagePullPolicy: Always
           resources:
             limits:
               cpu: 200m
               memory: 1000Mi
             requests:
               cpu: 50m
               memory: 500Mi
           ports:
           - name: prometheus
             containerPort: 8942
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:
               drop:
                 - ALL
             seccompProfile:
               type: RuntimeDefault
         serviceAccountName: alt-vpa-recommender-sa 3
         securityContext:
           runAsNonRoot: true

   1

   Creates a container for your alternative recommender.

   2

   Specifies your recommender image.

   3

   Associates the service account that you created for the recommender.

   A new pod is created for the alternative recommender in the same namespace.

   $ oc get pods

   Example output

   NAME                                        READY   STATUS    RESTARTS   AGE
   frontend-845d5478d-558zf                    1/1     Running   0          4m25s
   frontend-845d5478d-7z9gx                    1/1     Running   0          4m25s
   frontend-845d5478d-b7l4j                    1/1     Running   0          4m25s
   vpa-alt-recommender-55878867f9-6tp5v        1/1     Running   0          9s

3. Configure a VPA CR that includes the name of the alternative recommender Deployment object.

   Example VPA CR to include the alternative recommender

   apiVersion: autoscaling.k8s.io/v1
   kind: VerticalPodAutoscaler
   metadata:
     name: vpa-recommender
     namespace: <namespace_name>
   spec:
     recommenders:
       - name: alt-vpa-recommender 1
     targetRef:
       apiVersion: "apps/v1"
       kind:       Deployment 2
       name:       frontend

   1

   Specifies the name of the alternative recommender deployment.

   2

   Specifies the name of an existing workload object you want this VPA to manage.

Procedure

1. Create a Secret object in a YAML file on a control plane node.

   For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: mysecret
   type: Opaque 1
   data:
     username: <username>
     password: <password>

   1

   Specifies an opaque secret.

2. Use the following command to create a Secret object:

   $ oc create -f <filename>.yaml

3. To use the secret in a pod:

   1. Update the pod’s service account to reference the secret, as shown in the "Understanding how to create secrets" section.
   2. Create the pod, which consumes the secret as an environment variable or as a file (using a secret volume), as shown in the "Understanding how to create secrets" section.

Procedure

1. Create a Secret object in a YAML file on a control plane node:

   Example Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-sa-sample
     annotations:
       kubernetes.io/service-account.name: "sa-name" 1
   type: kubernetes.io/service-account-token 2

   1

   Specifies an existing service account name. If you are creating both the ServiceAccount and the Secret objects, create the ServiceAccount object first.

   2

   Specifies a service account token secret.

2. Use the following command to create the Secret object:

   $ oc create -f <filename>.yaml

3. To use the secret in a pod:

   1. Update the pod’s service account to reference the secret, as shown in the "Understanding how to create secrets" section.
   2. Create the pod, which consumes the secret as an environment variable or as a file (using a secret volume), as shown in the "Understanding how to create secrets" section.

Procedure

1. Create a Secret object in a YAML file on a control plane node:

   Example secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-basic-auth
   type: kubernetes.io/basic-auth 1
   data:
   stringData: 2
     username: admin
     password: <password>

   1

   Specifies a basic authentication secret.

   2

   Specifies the basic authentication values to use.

2. Use the following command to create the Secret object:

   $ oc create -f <filename>.yaml

3. To use the secret in a pod:

   1. Update the pod’s service account to reference the secret, as shown in the "Understanding how to create secrets" section.
   2. Create the pod, which consumes the secret as an environment variable or as a file (using a secret volume), as shown in the "Understanding how to create secrets" section.

Procedure

1. Create a Secret object in a YAML file on a control plane node:

   Example secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-ssh-auth
   type: kubernetes.io/ssh-auth 1
   data:
     ssh-privatekey: | 2
             MIIEpQIBAAKCAQEAulqb/Y ...

   1

   Specifies an SSH authentication secret.

   2

   Specifies the SSH key/value pair as the SSH credentials to use.

2. Use the following command to create the Secret object:

   $ oc create -f <filename>.yaml

3. To use the secret in a pod:

   1. Update the pod’s service account to reference the secret, as shown in the "Understanding how to create secrets" section.
   2. Create the pod, which consumes the secret as an environment variable or as a file (using a secret volume), as shown in the "Understanding how to create secrets" section.

Procedure

1. Create a Secret object in a YAML file on a control plane node.

   Example Docker configuration secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-docker-cfg
     namespace: my-project
   type: kubernetes.io/dockerconfig 1
   data:
     .dockerconfig:bm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg== 2

   1

   Specifies that the secret is using a Docker configuration file.

   2

   The output of a base64-encoded Docker configuration file

   Example Docker configuration JSON secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-docker-json
     namespace: my-project
   type: kubernetes.io/dockerconfig 1
   data:
     .dockerconfigjson:bm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg== 2

   1

   Specifies that the secret is using a Docker configuration JSONfile.

   2

   The output of a base64-encoded Docker configuration JSON file

2. Use the following command to create the Secret object

   $ oc create -f <filename>.yaml

3. To use the secret in a pod:

   1. Update the pod’s service account to reference the secret, as shown in the "Understanding how to create secrets" section.
   2. Create the pod, which consumes the secret as an environment variable or as a file (using a secret volume), as shown in the "Understanding how to create secrets" section.

Procedure

1. Navigate to Workloads Secrets.

2. Click Create From YAML.

   1. Edit the YAML manually to your specifications, or drag and drop a file into the YAML editor. For example:

      apiVersion: v1
      kind: Secret
      metadata:
        name: example
        namespace: <namespace>
      type: Opaque 1
      data:
        username: <base64 encoded username>
        password: <base64 encoded password>
      stringData: 2
        hostname: myapp.mydomain.com

      1

      This example specifies an opaque secret; however, you may see other secret types such as service account token secret, basic authentication secret, SSH authentication secret, or a secret that uses Docker configuration.

      2

      Entries in the stringData map are converted to base64 and the entry will then be moved to the data map automatically. This field is write-only; the value will only be returned via the data field.

3. Click Create.

4. Click Add Secret to workload.

   1. From the drop-down menu, select the workload to add.
   2. Click Save.

1. Edit the Pod spec for your service.

2. Add the service.beta.openshift.io/serving-cert-secret-name annotation with the name you want to use for your secret.

   kind: Service
   apiVersion: v1
   metadata:
     name: my-service
     annotations:
         service.beta.openshift.io/serving-cert-secret-name: my-cert 1
   spec:
     selector:
       app: MyApp
     ports:
     - protocol: TCP
       port: 80
       targetPort: 9376

   The certificate and key are in PEM format, stored in tls.crt and tls.key respectively.

3. Create the service:

   $ oc create -f <file-name>.yaml

4. View the secret to make sure it was created:

   1. View a list of all secrets:

      $ oc get secrets

      Example output

      NAME                     TYPE                                  DATA      AGE
      my-cert                  kubernetes.io/tls                     2         9m

   2. View details on your secret:

      $ oc describe secret my-cert

      Example output

      Name:         my-cert
      Namespace:    openshift-console
      Labels:       <none>
      Annotations:  service.beta.openshift.io/expiry: 2023-03-08T23:22:40Z
                    service.beta.openshift.io/originating-service-name: my-service
                    service.beta.openshift.io/originating-service-uid: 640f0ec3-afc2-4380-bf31-a8c784846a11
                    service.beta.openshift.io/expiry: 2023-03-08T23:22:40Z
      
      Type:  kubernetes.io/tls
      
      Data
      ====
      tls.key:  1679 bytes
      tls.crt:  2595 bytes

5. Edit your Pod spec with that secret.

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-service-pod
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: mypod
       image: redis
       volumeMounts:
       - name: my-container
         mountPath: "/etc/my-path"
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]
     volumes:
     - name: my-volume
       secret:
         secretName: my-cert
         items:
         - key: username
           path: my-group/my-username
           mode: 511

   When it is available, your pod will run. The certificate will be good for the internal service DNS name, <service.name>.<service.namespace>.svc.

   The certificate/key pair is automatically replaced when it gets close to expiration. View the expiration date in the service.beta.openshift.io/expiry annotation on the secret, which is in RFC3339 format.

   Note

   In most cases, the service DNS name <service.name>.<service.namespace>.svc is not externally routable. The primary use of <service.name>.<service.namespace>.svc is for intracluster or intraservice communication, and with re-encrypt routes.

Procedure

1. Install the AWS Secrets Manager provider:

   1. Create a YAML file with the following configuration for the provider resources:

      Important

      The AWS Secrets Manager provider for the Secrets Store CSI driver is an upstream provider.

      This configuration is modified from the configuration provided in the upstream AWS documentation so that it works properly with OpenShift Container Platform. Changes to this configuration might impact functionality.

      Example aws-provider.yaml file

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: csi-secrets-store-provider-aws
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: csi-secrets-store-provider-aws-cluster-role
      rules:
      - apiGroups: [""]
        resources: ["serviceaccounts/token"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["serviceaccounts"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: csi-secrets-store-provider-aws-cluster-rolebinding
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: csi-secrets-store-provider-aws-cluster-role
      subjects:
      - kind: ServiceAccount
        name: csi-secrets-store-provider-aws
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        namespace: openshift-cluster-csi-drivers
        name: csi-secrets-store-provider-aws
        labels:
          app: csi-secrets-store-provider-aws
      spec:
        updateStrategy:
          type: RollingUpdate
        selector:
          matchLabels:
            app: csi-secrets-store-provider-aws
        template:
          metadata:
            labels:
              app: csi-secrets-store-provider-aws
          spec:
            serviceAccountName: csi-secrets-store-provider-aws
            hostNetwork: false
            containers:
              - name: provider-aws-installer
                image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-50-g5b4aca1-2023.06.09.21.19
                imagePullPolicy: Always
                args:
                    - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
                resources:
                  requests:
                    cpu: 50m
                    memory: 100Mi
                  limits:
                    cpu: 50m
                    memory: 100Mi
                securityContext:
                  privileged: true
                volumeMounts:
                  - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
                    name: providervol
                  - name: mountpoint-dir
                    mountPath: /var/lib/kubelet/pods
                    mountPropagation: HostToContainer
            tolerations:
            - operator: Exists
            volumes:
              - name: providervol
                hostPath:
                  path: "/etc/kubernetes/secrets-store-csi-providers"
              - name: mountpoint-dir
                hostPath:
                  path: /var/lib/kubelet/pods
                  type: DirectoryOrCreate
            nodeSelector:
              kubernetes.io/os: linux

   2. Grant privileged access to the csi-secrets-store-provider-aws service account by running the following command:

      $ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-aws -n openshift-cluster-csi-drivers

   3. Create the provider resources by running the following command:

      $ oc apply -f aws-provider.yaml

2. Grant permission to allow the service account to read the AWS secret object:

   1. Create a directory to contain the credentials request by running the following command:

      $ mkdir credentialsrequest-dir-aws

   2. Create a YAML file with the following configuration for the credentials request:

      Example credentialsrequest.yaml file

      apiVersion: cloudcredential.openshift.io/v1
      kind: CredentialsRequest
      metadata:
        name: aws-provider-test
        namespace: openshift-cloud-credential-operator
      spec:
        providerSpec:
          apiVersion: cloudcredential.openshift.io/v1
          kind: AWSProviderSpec
          statementEntries:
          - action:
            - "secretsmanager:GetSecretValue"
            - "secretsmanager:DescribeSecret"
            effect: Allow
            resource: "arn:*:secretsmanager:*:*:secret:testSecret-??????"
        secretRef:
          name: aws-creds
          namespace: my-namespace
        serviceAccountNames:
        - aws-provider

   3. Retrieve the OIDC provider by running the following command:

      $ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'

      Example output

      https://<oidc_provider_name>

      Copy the OIDC provider name <oidc_provider_name> from the output to use in the next step.

   4. Use the ccoctl tool to process the credentials request by running the following command:

      $ ccoctl aws create-iam-roles \
          --name my-role --region=<aws_region> \
          --credentials-requests-dir=credentialsrequest-dir-aws \
          --identity-provider-arn arn:aws:iam::<aws_account>:oidc-provider/<oidc_provider_name> --output-dir=credrequests-ccoctl-output

      Example output

      2023/05/15 18:10:34 Role arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds created
      2023/05/15 18:10:34 Saved credentials configuration to: credrequests-ccoctl-output/manifests/my-namespace-aws-creds-credentials.yaml
      2023/05/15 18:10:35 Updated Role policy for Role my-role-my-namespace-aws-creds

      Copy the <aws_role_arn> from the output to use in the next step. For example, arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds.

   5. Bind the service account with the role ARN by running the following command:

      $ oc annotate -n my-namespace sa/aws-provider eks.amazonaws.com/role-arn="<aws_role_arn>"

3. Create a secret provider class to define your secrets store provider:

   1. Create a YAML file that defines the SecretProviderClass object:

      Example secret-provider-class-aws.yaml

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: my-aws-provider                   1
        namespace: my-namespace                 2
      spec:
        provider: aws                           3
        parameters:                             4
          objects: |
            - objectName: "testSecret"
              objectType: "secretsmanager"

      1 1

      Specify the name for the secret provider class.

      2

      Specify the namespace for the secret provider class.

      3

      Specify the provider as aws.

      4

      Specify the provider-specific configuration parameters.

   2. Create the SecretProviderClass object by running the following command:

      $ oc create -f secret-provider-class-aws.yaml

4. Create a deployment to use this secret provider class:

   1. Create a YAML file that defines the Deployment object:

      Example deployment.yaml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: my-aws-deployment                              1
        namespace: my-namespace                              2
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: my-storage
        template:
          metadata:
            labels:
              app: my-storage
          spec:
            serviceAccountName: aws-provider
            containers:
            - name: busybox
              image: k8s.gcr.io/e2e-test-images/busybox:1.29
              command:
                - "/bin/sleep"
                - "10000"
              volumeMounts:
              - name: secrets-store-inline
                mountPath: "/mnt/secrets-store"
                readOnly: true
            volumes:
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: "my-aws-provider" 3

      1

      Specify the name for the deployment.

      2

      Specify the namespace for the deployment. This must be the same namespace as the secret provider class.

      3

      Specify the name of the secret provider class.

   2. Create the Deployment object by running the following command:

      $ oc create -f deployment.yaml

Procedure

1. Install the AWS Systems Manager Parameter Store provider:

   1. Create a YAML file with the following configuration for the provider resources:

      Important

      The AWS Systems Manager Parameter Store provider for the Secrets Store CSI driver is an upstream provider.

      This configuration is modified from the configuration provided in the upstream AWS documentation so that it works properly with OpenShift Container Platform. Changes to this configuration might impact functionality.

      Example aws-provider.yaml file

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: csi-secrets-store-provider-aws
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: csi-secrets-store-provider-aws-cluster-role
      rules:
      - apiGroups: [""]
        resources: ["serviceaccounts/token"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["serviceaccounts"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: csi-secrets-store-provider-aws-cluster-rolebinding
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: csi-secrets-store-provider-aws-cluster-role
      subjects:
      - kind: ServiceAccount
        name: csi-secrets-store-provider-aws
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        namespace: openshift-cluster-csi-drivers
        name: csi-secrets-store-provider-aws
        labels:
          app: csi-secrets-store-provider-aws
      spec:
        updateStrategy:
          type: RollingUpdate
        selector:
          matchLabels:
            app: csi-secrets-store-provider-aws
        template:
          metadata:
            labels:
              app: csi-secrets-store-provider-aws
          spec:
            serviceAccountName: csi-secrets-store-provider-aws
            hostNetwork: false
            containers:
              - name: provider-aws-installer
                image: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-50-g5b4aca1-2023.06.09.21.19
                imagePullPolicy: Always
                args:
                    - --provider-volume=/etc/kubernetes/secrets-store-csi-providers
                resources:
                  requests:
                    cpu: 50m
                    memory: 100Mi
                  limits:
                    cpu: 50m
                    memory: 100Mi
                securityContext:
                  privileged: true
                volumeMounts:
                  - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
                    name: providervol
                  - name: mountpoint-dir
                    mountPath: /var/lib/kubelet/pods
                    mountPropagation: HostToContainer
            tolerations:
            - operator: Exists
            volumes:
              - name: providervol
                hostPath:
                  path: "/etc/kubernetes/secrets-store-csi-providers"
              - name: mountpoint-dir
                hostPath:
                  path: /var/lib/kubelet/pods
                  type: DirectoryOrCreate
            nodeSelector:
              kubernetes.io/os: linux

   2. Grant privileged access to the csi-secrets-store-provider-aws service account by running the following command:

      $ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-aws -n openshift-cluster-csi-drivers

   3. Create the provider resources by running the following command:

      $ oc apply -f aws-provider.yaml

2. Grant permission to allow the service account to read the AWS secret object:

   1. Create a directory to contain the credentials request by running the following command:

      $ mkdir credentialsrequest-dir-aws

   2. Create a YAML file with the following configuration for the credentials request:

      Example credentialsrequest.yaml file

      apiVersion: cloudcredential.openshift.io/v1
      kind: CredentialsRequest
      metadata:
        name: aws-provider-test
        namespace: openshift-cloud-credential-operator
      spec:
        providerSpec:
          apiVersion: cloudcredential.openshift.io/v1
          kind: AWSProviderSpec
          statementEntries:
          - action:
            - "ssm:GetParameter"
            - "ssm:GetParameters"
            effect: Allow
            resource: "arn:*:ssm:*:*:parameter/testParameter*"
        secretRef:
          name: aws-creds
          namespace: my-namespace
        serviceAccountNames:
        - aws-provider

   3. Retrieve the OIDC provider by running the following command:

      $ oc get --raw=/.well-known/openid-configuration | jq -r '.issuer'

      Example output

      https://<oidc_provider_name>

      Copy the OIDC provider name <oidc_provider_name> from the output to use in the next step.

   4. Use the ccoctl tool to process the credentials request by running the following command:

      $ ccoctl aws create-iam-roles \
          --name my-role --region=<aws_region> \
          --credentials-requests-dir=credentialsrequest-dir-aws \
          --identity-provider-arn arn:aws:iam::<aws_account>:oidc-provider/<oidc_provider_name> --output-dir=credrequests-ccoctl-output

      Example output

      2023/05/15 18:10:34 Role arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds created
      2023/05/15 18:10:34 Saved credentials configuration to: credrequests-ccoctl-output/manifests/my-namespace-aws-creds-credentials.yaml
      2023/05/15 18:10:35 Updated Role policy for Role my-role-my-namespace-aws-creds

      Copy the <aws_role_arn> from the output to use in the next step. For example, arn:aws:iam::<aws_account_id>:role/my-role-my-namespace-aws-creds.

   5. Bind the service account with the role ARN by running the following command:

      $ oc annotate -n my-namespace sa/aws-provider eks.amazonaws.com/role-arn="<aws_role_arn>"

3. Create a secret provider class to define your secrets store provider:

   1. Create a YAML file that defines the SecretProviderClass object:

      Example secret-provider-class-aws.yaml

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: my-aws-provider                   1
        namespace: my-namespace                 2
      spec:
        provider: aws                           3
        parameters:                             4
          objects: |
            - objectName: "testParameter"
              objectType: "ssmparameter"

      1

      Specify the name for the secret provider class.

      2

      Specify the namespace for the secret provider class.

      3

      Specify the provider as aws.

      4

      Specify the provider-specific configuration parameters.

   2. Create the SecretProviderClass object by running the following command:

      $ oc create -f secret-provider-class-aws.yaml

4. Create a deployment to use this secret provider class:

   1. Create a YAML file that defines the Deployment object:

      Example deployment.yaml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: my-aws-deployment                              1
        namespace: my-namespace                              2
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: my-storage
        template:
          metadata:
            labels:
              app: my-storage
          spec:
            serviceAccountName: aws-provider
            containers:
            - name: busybox
              image: k8s.gcr.io/e2e-test-images/busybox:1.29
              command:
                - "/bin/sleep"
                - "10000"
              volumeMounts:
              - name: secrets-store-inline
                mountPath: "/mnt/secrets-store"
                readOnly: true
            volumes:
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: "my-aws-provider" 3

      1

      Specify the name for the deployment.

      2

      Specify the namespace for the deployment. This must be the same namespace as the secret provider class.

      3

      Specify the name of the secret provider class.

   2. Create the Deployment object by running the following command:

      $ oc create -f deployment.yaml

Procedure

1. Install the Azure Key Vault provider:

   1. Create a YAML file with the following configuration for the provider resources:

      Important

      The Azure Key Vault provider for the Secrets Store CSI driver is an upstream provider.

      This configuration is modified from the configuration provided in the upstream Azure documentation so that it works properly with OpenShift Container Platform. Changes to this configuration might impact functionality.

      Example azure-provider.yaml file

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: csi-secrets-store-provider-azure
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: csi-secrets-store-provider-azure-cluster-role
      rules:
      - apiGroups: [""]
        resources: ["serviceaccounts/token"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["serviceaccounts"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: csi-secrets-store-provider-azure-cluster-rolebinding
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: csi-secrets-store-provider-azure-cluster-role
      subjects:
      - kind: ServiceAccount
        name: csi-secrets-store-provider-azure
        namespace: openshift-cluster-csi-drivers
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        namespace: openshift-cluster-csi-drivers
        name: csi-secrets-store-provider-azure
        labels:
          app: csi-secrets-store-provider-azure
      spec:
        updateStrategy:
          type: RollingUpdate
        selector:
          matchLabels:
            app: csi-secrets-store-provider-azure
        template:
          metadata:
            labels:
              app: csi-secrets-store-provider-azure
          spec:
            serviceAccountName: csi-secrets-store-provider-azure
            hostNetwork: true
            containers:
              - name: provider-azure-installer
                image: mcr.microsoft.com/oss/azure/secrets-store/provider-azure:v1.4.1
                imagePullPolicy: IfNotPresent
                args:
                  - --endpoint=unix:///provider/azure.sock
                  - --construct-pem-chain=true
                  - --healthz-port=8989
                  - --healthz-path=/healthz
                  - --healthz-timeout=5s
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 8989
                  failureThreshold: 3
                  initialDelaySeconds: 5
                  timeoutSeconds: 10
                  periodSeconds: 30
                resources:
                  requests:
                    cpu: 50m
                    memory: 100Mi
                  limits:
                    cpu: 50m
                    memory: 100Mi
                securityContext:
                  allowPrivilegeEscalation: false
                  readOnlyRootFilesystem: true
                  runAsUser: 0
                  capabilities:
                    drop:
                    - ALL
                volumeMounts:
                  - mountPath: "/provider"
                    name: providervol
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
            volumes:
              - name: providervol
                hostPath:
                  path: "/var/run/secrets-store-csi-providers"
            tolerations:
            - operator: Exists
            nodeSelector:
              kubernetes.io/os: linux

   2. Grant privileged access to the csi-secrets-store-provider-azure service account by running the following command:

      $ oc adm policy add-scc-to-user privileged -z csi-secrets-store-provider-azure -n openshift-cluster-csi-drivers

   3. Create the provider resources by running the following command:

      $ oc apply -f azure-provider.yaml

2. Create a service principal to access the key vault:

   1. Set the service principal client secret as an environment variable by running the following command:

      $ SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --name https://$KEYVAULT_NAME --query 'password' -otsv)"

   2. Set the service principal client ID as an environment variable by running the following command:

      $ SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name https://$KEYVAULT_NAME --query '[0].appId' -otsv)"

   3. Create a generic secret with the service principal client secret and ID by running the following command:

      $ oc create secret generic secrets-store-creds -n my-namespace --from-literal clientid=${SERVICE_PRINCIPAL_CLIENT_ID} --from-literal clientsecret=${SERVICE_PRINCIPAL_CLIENT_SECRET}

   4. Apply the secrets-store.csi.k8s.io/used=true label to allow the provider to find this nodePublishSecretRef secret:

      $ oc -n my-namespace label secret secrets-store-creds secrets-store.csi.k8s.io/used=true

3. Create a secret provider class to define your secrets store provider:

   1. Create a YAML file that defines the SecretProviderClass object:

      Example secret-provider-class-azure.yaml

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: my-azure-provider                 1
        namespace: my-namespace                 2
      spec:
        provider: azure                         3
        parameters:                             4
          usePodIdentity: "false"
          useVMManagedIdentity: "false"
          userAssignedIdentityID: ""
          keyvaultName: "kvname"
          objects: |
            array:
              - |
                objectName: secret1
                objectType: secret
          tenantId: "tid"

      1

      Specify the name for the secret provider class.

      2

      Specify the namespace for the secret provider class.

      3

      Specify the provider as azure.

      4

      Specify the provider-specific configuration parameters.

   2. Create the SecretProviderClass object by running the following command:

      $ oc create -f secret-provider-class-azure.yaml

4. Create a deployment to use this secret provider class:

   1. Create a YAML file that defines the Deployment object:

      Example deployment.yaml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: my-azure-deployment                            1
        namespace: my-namespace                              2
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: my-storage
        template:
          metadata:
            labels:
              app: my-storage
          spec:
            containers:
            - name: busybox
              image: k8s.gcr.io/e2e-test-images/busybox:1.29
              command:
                - "/bin/sleep"
                - "10000"
              volumeMounts:
              - name: secrets-store-inline
                mountPath: "/mnt/secrets-store"
                readOnly: true
            volumes:
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: "my-azure-provider" 3
                  nodePublishSecretRef:
                    name: secrets-store-creds                4

      1

      Specify the name for the deployment.

      2

      Specify the namespace for the deployment. This must be the same namespace as the secret provider class.

      3

      Specify the name of the secret provider class.

      4

      Specify the name of the Kubernetes secret that contains the service principal credentials to access Azure Key Vault.

   2. Create the Deployment object by running the following command:

      $ oc create -f deployment.yaml

Procedure

1. Add the HashiCorp Helm repository by running the following command:

   $ helm repo add hashicorp https://helm.releases.hashicorp.com

2. Update all repositories to ensure that Helm is aware of the latest versions by running the following command:

   $ helm repo update

3. Install the HashiCorp Vault provider:

   1. Create a new project for Vault by running the following command:

      $ oc new-project vault

   2. Label the vault namespace for pod security admission by running the following command:

      $ oc label ns vault security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/audit=privileged pod-security.kubernetes.io/warn=privileged --overwrite

   3. Grant privileged access to the vault service account by running the following command:

      $ oc adm policy add-scc-to-user privileged -z vault -n vault

   4. Grant privileged access to the vault-csi-provider service account by running the following command:

      $ oc adm policy add-scc-to-user privileged -z vault-csi-provider -n vault

   5. Deploy HashiCorp Vault by running the following command:

      $ helm install vault hashicorp/vault --namespace=vault \
        --set "server.dev.enabled=true" \
        --set "injector.enabled=false" \
        --set "csi.enabled=true" \
        --set "global.openshift=true" \
        --set "injector.agentImage.repository=docker.io/hashicorp/vault" \
        --set "server.image.repository=docker.io/hashicorp/vault" \
        --set "csi.image.repository=docker.io/hashicorp/vault-csi-provider" \
        --set "csi.agent.image.repository=docker.io/hashicorp/vault" \
        --set "csi.daemonSet.providersDir=/var/run/secrets-store-csi-providers"

   6. Patch the vault-csi-driver daemon set to set the securityContext to privileged by running the following command:

      $ oc patch daemonset -n vault vault-csi-provider --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": true} }]'

   7. Verify that the vault-csi-provider pods have started properly by running the following command:

      $ oc get pods -n vault

      Example output

      NAME                       READY   STATUS    RESTARTS   AGE
      vault-0                    1/1     Running   0          24m
      vault-csi-provider-87rgw   1/2     Running   0          5s
      vault-csi-provider-bd6hp   1/2     Running   0          4s
      vault-csi-provider-smlv7   1/2     Running   0          5s

4. Configure HashiCorp Vault to store the required secrets:

   1. Create a secret by running the following command:

      $ oc exec vault-0 --namespace=vault -- vault kv put secret/example1 testSecret1=my-secret-value

   2. Verify that the secret is readable at the path secret/example1 by running the following command:

      $ oc exec vault-0 --namespace=vault -- vault kv get secret/example1

      Example output

      = Secret Path =
      secret/data/example1
      
      ======= Metadata =======
      Key                Value
      ---                -----
      created_time       2024-04-05T07:05:16.713911211Z
      custom_metadata    <nil>
      deletion_time      n/a
      destroyed          false
      version            1
      
      === Data ===
      Key            Value
      ---            -----
      testSecret1    my-secret-value

5. Configure Vault to use Kubernetes authentication:

   1. Enable the Kubernetes auth method by running the following command:

      $ oc exec vault-0 --namespace=vault -- vault auth enable kubernetes

      Example output

      Success! Enabled kubernetes auth method at: kubernetes/

   2. Configure the Kubernetes auth method:

      1. Set the token reviewer as an environment variable by running the following command:

         $ TOKEN_REVIEWER_JWT="$(oc exec vault-0 --namespace=vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

      2. Set the Kubernetes service IP address as an environment variable by running the following command:

         $ KUBERNETES_SERVICE_IP="$(oc get svc kubernetes -o go-template="{{ .spec.clusterIP }}")"

      3. Update the Kubernetes auth method by running the following command:

         $ oc exec -i vault-0 --namespace=vault -- vault write auth/kubernetes/config \
           issuer="https://kubernetes.default.svc.cluster.local" \
           token_reviewer_jwt="${TOKEN_REVIEWER_JWT}" \
           kubernetes_host="https://${KUBERNETES_SERVICE_IP}:443" \
           kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

         Example output

         Success! Data written to: auth/kubernetes/config

   3. Create a policy for the application by running the following command:

      $ oc exec -i vault-0 --namespace=vault -- vault policy write csi -<<EOF
        path "secret/data/*" {
        capabilities = ["read"]
        }
        EOF

      Example output

      Success! Uploaded policy: csi

   4. Create an authentication role to access the application by running the following command:

      $ oc exec -i vault-0 --namespace=vault -- vault write auth/kubernetes/role/csi \
        bound_service_account_names=default \
        bound_service_account_namespaces=default,test-ns,negative-test-ns,my-namespace \
        policies=csi \
        ttl=20m

      Example output

      Success! Data written to: auth/kubernetes/role/csi

   5. Verify that all of the vault pods are running properly by running the following command:

      $ oc get pods -n vault

      Example output

      NAME                       READY   STATUS    RESTARTS   AGE
      vault-0                    1/1     Running   0          43m
      vault-csi-provider-87rgw   2/2     Running   0          19m
      vault-csi-provider-bd6hp   2/2     Running   0          19m
      vault-csi-provider-smlv7   2/2     Running   0          19m

   6. Verify that all of the secrets-store-csi-driver pods are running properly by running the following command:

      $ oc get pods -n openshift-cluster-csi-drivers | grep -E "secrets"

      Example output

      secrets-store-csi-driver-node-46d2g                  3/3     Running   0             45m
      secrets-store-csi-driver-node-d2jjn                  3/3     Running   0             45m
      secrets-store-csi-driver-node-drmt4                  3/3     Running   0             45m
      secrets-store-csi-driver-node-j2wlt                  3/3     Running   0             45m
      secrets-store-csi-driver-node-v9xv4                  3/3     Running   0             45m
      secrets-store-csi-driver-node-vlz28                  3/3     Running   0             45m
      secrets-store-csi-driver-operator-84bd699478-fpxrw   1/1     Running   0             47m

6. Create a secret provider class to define your secrets store provider:

   1. Create a YAML file that defines the SecretProviderClass object:

      Example secret-provider-class-vault.yaml

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: my-vault-provider                   1
        namespace: my-namespace                   2
      spec:
        provider: vault                           3
        parameters:                               4
          roleName: "csi"
          vaultAddress: "http://vault.vault:8200"
          objects:  |
            - secretPath: "secret/data/example1"
              objectName: "testSecret1"
              secretKey: "testSecret1

      1

      Specify the name for the secret provider class.

      2

      Specify the namespace for the secret provider class.

      3

      Specify the provider as vault.

      4

      Specify the provider-specific configuration parameters.

   2. Create the SecretProviderClass object by running the following command:

      $ oc create -f secret-provider-class-vault.yaml

7. Create a deployment to use this secret provider class:

   1. Create a YAML file that defines the Deployment object:

      Example deployment.yaml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: busybox-deployment                                    1
        namespace: my-namespace                                     2
        labels:
          app: busybox
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: busybox
        template:
          metadata:
            labels:
              app: busybox
          spec:
            terminationGracePeriodSeconds: 0
            containers:
            - image: registry.k8s.io/e2e-test-images/busybox:1.29-4
              name: busybox
              imagePullPolicy: IfNotPresent
              command:
              - "/bin/sleep"
              - "10000"
              volumeMounts:
              - name: secrets-store-inline
                mountPath: "/mnt/secrets-store"
                readOnly: true
            volumes:
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: "my-vault-provider"        3

      1

      Specify the name for the deployment.

      2

      Specify the namespace for the deployment. This must be the same namespace as the secret provider class.

      3

      Specify the name of the secret provider class.

   2. Create the Deployment object by running the following command:

      $ oc create -f deployment.yaml

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Create the required namespace for the Run Once Duration Override Operator.

   1. Navigate to Administration Namespaces and click Create Namespace.
   2. Enter openshift-run-once-duration-override-operator in the Name field and click Create.

3. Install the Run Once Duration Override Operator.

   1. Navigate to Operators OperatorHub.
   2. Enter Run Once Duration Override Operator into the filter box.
   3. Select the Run Once Duration Override Operator and click Install.

   4. On the Install Operator page:

      1. The Update channel is set to stable, which installs the latest stable release of the Run Once Duration Override Operator.
      2. Select A specific namespace on the cluster.
      3. Choose openshift-run-once-duration-override-operator from the dropdown menu under Installed namespace.

      4. Select an Update approval strategy.

         • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
         • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
      5. Click Install.

4. Create a RunOnceDurationOverride instance.

   1. From the Operators Installed Operators page, click Run Once Duration Override Operator.
   2. Select the Run Once Duration Override tab and click Create RunOnceDurationOverride.

   3. Edit the settings as necessary.

      Under the runOnceDurationOverride section, you can update the spec.activeDeadlineSeconds value, if required. The predefined value is 3600 seconds, or 1 hour.

   4. Click Create.

Verification

1. Log in to the OpenShift CLI.

2. Verify all pods are created and running properly.

   $ oc get pods -n openshift-run-once-duration-override-operator

   Example output

   NAME                                                   READY   STATUS    RESTARTS   AGE
   run-once-duration-override-operator-7b88c676f6-lcxgc   1/1     Running   0          7m46s
   runoncedurationoverride-62blp                          1/1     Running   0          41s
   runoncedurationoverride-h8h8b                          1/1     Running   0          41s
   runoncedurationoverride-tdsqk                          1/1     Running   0          41s

Procedure

1. Log in to the OpenShift CLI.

2. Add the label to enable the run-once duration override to your namespace:

   $ oc label namespace <namespace> \ 1
       runoncedurationoverrides.admission.runoncedurationoverride.openshift.io/enabled=true

   1

   Specify the namespace to enable the run-once duration override on.

Verification

1. Create a test run-once pod in the namespace that you enabled the run-once duration override on:

   apiVersion: v1
   kind: Pod
   metadata:
     name: example
     namespace: <namespace>                 1
   spec:
     restartPolicy: Never                   2
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
       - name: busybox
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop: [ALL]
         image: busybox:1.25
         command:
           - /bin/sh
           - -ec
           - |
             while sleep 5; do date; done

   1

   Replace <namespace> with the name of your namespace.

   2

   The restartPolicy must be Never or OnFailure to be a run-once pod.

2. Verify that the pod has its activeDeadlineSeconds field set:

   $ oc get pods -n <namespace> -o yaml | grep activeDeadlineSeconds

   Example output

       activeDeadlineSeconds: 3600

Procedure

1. Log in to the OpenShift CLI.

2. Edit the RunOnceDurationOverride resource:

   $ oc edit runoncedurationoverride cluster

3. Update the activeDeadlineSeconds field:

   apiVersion: operator.openshift.io/v1
   kind: RunOnceDurationOverride
   metadata:
   # ...
   spec:
     runOnceDurationOverride:
       spec:
         activeDeadlineSeconds: 1800 1
   # ...

   1

   Set the activeDeadlineSeconds field to the desired value, in seconds.

4. Save the file to apply the changes.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators Installed Operators.
3. Select openshift-run-once-duration-override-operator from the Project dropdown list.

4. Delete the RunOnceDurationOverride instance.

   1. Click Run Once Duration Override Operator and select the Run Once Duration Override tab.
   2. Click the Options menu next to the cluster entry and select Delete RunOnceDurationOverride.
   3. In the confirmation dialog, click Delete.

5. Uninstall the Run Once Duration Override Operator Operator.

   1. Navigate to Operators Installed Operators.
   2. Click the Options menu next to the Run Once Duration Override Operator entry and click Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Remove CRDs that were created when the Run Once Duration Override Operator was installed:

   1. Navigate to Administration CustomResourceDefinitions.
   2. Enter RunOnceDurationOverride in the Name field to filter the CRDs.
   3. Click the Options menu next to the RunOnceDurationOverride CRD and select Delete CustomResourceDefinition.
   4. In the confirmation dialog, click Delete.

3. Delete the openshift-run-once-duration-override-operator namespace.

   1. Navigate to Administration Namespaces.
   2. Enter openshift-run-once-duration-override-operator into the filter box.
   3. Click the Options menu next to the openshift-run-once-duration-override-operator entry and select Delete Namespace.
   4. In the confirmation dialog, enter openshift-run-once-duration-override-operator and click Delete.

4. Remove the run-once duration override label from the namespaces that it was enabled on.

   1. Navigate to Administration Namespaces.
   2. Select your namespace.
   3. Click Edit next to the Labels field.
   4. Remove the runoncedurationoverrides.admission.runoncedurationoverride.openshift.io/enabled=true label and click Save.