Chapter 19. Managing cloud provider credentials

Procedure

1. Log in to the OpenShift Container Platform web console as a user with the

   cluster-admin

   role.
2. Navigate to Administration Cluster Settings.
3. On the Cluster Settings page, select the Configuration tab.
4. Under Configuration resource, select CloudCredential.
5. On the CloudCredential details page, select the YAML tab.

6. In the YAML block, check the value of

   spec.credentialsMode

   . The following values are possible, though not all are supported on all platforms:

   • ''

     : The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.

   • Mint

     : The CCO is operating in mint mode.

   • Passthrough

     : The CCO is operating in passthrough mode.

   • Manual

     : The CCO is operating in manual mode.
   Important

   To determine the specific configuration of an AWS or GCP cluster that has a

   spec.credentialsMode

   of

   ''

   ,

   Mint

   , or

   Manual

   , you must investigate further.

   AWS and GCP clusters support using mint mode with the root secret deleted.

   An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster

   Authentication

   object.

7. AWS or GCP clusters that use the default (

   ''

   ) only: To determine whether the cluster is operating in mint or passthrough mode, inspect the annotations on the cluster root secret:

   1. Navigate to Workloads Secrets and look for the root secret for your cloud provider.

      Note

      Ensure that the Project dropdown is set to All Projects.

      Expand

      Platform	Secret name
      AWS	aws-creds
      GCP	gcp-credentials

      Show more

   2. To view the CCO mode that the cluster is using, click

      1 annotation

      under Annotations, and check the value field. The following values are possible:

      • Mint

        : The CCO is operating in mint mode.

      • Passthrough

        : The CCO is operating in passthrough mode.

      If your cluster uses mint mode, you can also determine whether the cluster is operating without the root secret.

8. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, navigate to Workloads Secrets and look for the root secret for your cloud provider.

   Note

   Ensure that the Project dropdown is set to All Projects.

   Expand

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

   Show more
   • If you see one of these values, your cluster is using mint or passthrough mode with the root secret present.
   • If you do not see these values, your cluster is using the CCO in mint mode with the root secret removed.

9. AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, you must check the cluster

   Authentication

   object YAML values.
   1. Navigate to Administration Cluster Settings.
   2. On the Cluster Settings page, select the Configuration tab.
   3. Under Configuration resource, select Authentication.
   4. On the Authentication details page, select the YAML tab.

   5. In the YAML block, check the value of the

      .spec.serviceAccountIssuer

      parameter.
      • A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the

        ccoctl

        utility.
      • An empty value (

        ''

        ) indicates that the cluster is using the CCO in manual mode but was not configured using the

        ccoctl

        utility.

Procedure

1. Log in to

   oc

   on the cluster as a user with the

   cluster-admin

   role.

2. To determine the mode that the CCO is configured to use, enter the following command:

   $ oc get cloudcredentials cluster \
     -o=jsonpath={.spec.credentialsMode}

   The following output values are possible, though not all are supported on all platforms:

   • ''

     : The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.

   • Mint

     : The CCO is operating in mint mode.

   • Passthrough

     : The CCO is operating in passthrough mode.

   • Manual

     : The CCO is operating in manual mode.
   Important

   To determine the specific configuration of an AWS or GCP cluster that has a

   spec.credentialsMode

   of

   ''

   ,

   Mint

   , or

   Manual

   , you must investigate further.

   AWS and GCP clusters support using mint mode with the root secret deleted.

   An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster

   Authentication

   object.

3. AWS or GCP clusters that use the default (

   ''

   ) only: To determine whether the cluster is operating in mint or passthrough mode, run the following command:

   $ oc get secret <secret_name> \
     -n kube-system \
     -o jsonpath \
     --template '{ .metadata.annotations }'

   where

   <secret_name>

   is

   aws-creds

   for AWS or

   gcp-credentials

   for GCP.

   This command displays the value of the

   .metadata.annotations

   parameter in the cluster root secret object. The following output values are possible:

   • Mint

     : The CCO is operating in mint mode.

   • Passthrough

     : The CCO is operating in passthrough mode.

   If your cluster uses mint mode, you can also determine whether the cluster is operating without the root secret.

4. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, run the following command:

   $ oc get secret <secret_name> \
     -n=kube-system

   where

   <secret_name>

   is

   aws-creds

   for AWS or

   gcp-credentials

   for GCP.

   If the root secret is present, the output of this command returns information about the secret. An error indicates that the root secret is not present on the cluster.

5. AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, run the following command:

   $ oc get authentication cluster \
     -o jsonpath \
     --template='{ .spec.serviceAccountIssuer }'

   This command displays the value of the

   .spec.serviceAccountIssuer

   parameter in the cluster

   Authentication

   object.
   • An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the

     ccoctl

     utility.
   • An empty output indicates that the cluster is using the CCO in manual mode but was not configured using the

     ccoctl

     utility.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

   Show more
3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. Delete each component secret that is referenced by the individual

   CredentialsRequest

   objects.
   1. Log in to the OpenShift Container Platform CLI as a user with the

      cluster-admin

      role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where

      <provider_spec>

      is the corresponding value for your cloud provider:
      • AWS:

        AWSProviderSpec

      • GCP:

        GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \

      1

      
        -n <secret_namespace> 

      2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

   Show more
3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   RHV	ovirt-credentials
   VMware vSphere	vsphere-creds

   Show more
3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the

   cluster-admin

   role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports

   Progressing=true

   . To view the status, run the following command:

   $ oc get co kube-controller-manager

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. Log in to

   oc

   on the cluster as a user with the

   cluster-admin

   role.

2. Edit the

   CloudCredential

   resource to add an

   upgradeable-to

   annotation within the

   metadata

   field by running the following command:

   $ oc edit cloudcredential cluster

   Text to add

   ...
     metadata:
       annotations:
         cloudcredential.openshift.io/upgradeable-to: <version_number>
   ...

   Where

   <version_number>

   is the version that you are upgrading to, in the format

   x.y.z

   . For example, use

   4.12.2

   for OpenShift Container Platform 4.12.2.

   It may take several minutes after adding the annotation for the upgradeable status to change.

Verification

1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.

2. To view the CCO status details, click cloud-credential in the Cluster Operators list.

   • If the Upgradeable status in the Conditions section is False, verify that the

     upgradeable-to

     annotation is free of typographical errors.
3. When the Upgradeable status in the Conditions section is True, begin the OpenShift Container Platform upgrade.

Procedure

1. Obtain the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Note

   Ensure that the architecture of the

   $RELEASE_IMAGE

   matches the architecture of the environment in which you will use the

   ccoctl

   tool.

3. Extract the

   ccoctl

   binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

4. Change the permissions to make

   ccoctl

   executable by running the following command:

   $ chmod 775 ccoctl

Procedure

1. Change to the directory that contains the installation program and create the

   install-config.yaml

   file:

   $ openshift-install create install-config --dir <installation_directory>

   where

   <installation_directory>

   is the directory in which the installation program creates files.

2. Edit the

   install-config.yaml

   configuration file so that it contains the

   credentialsMode

   parameter set to

   Manual

   .

   Example install-config.yaml configuration file

   apiVersion: v1
   baseDomain: cluster1.example.com
   credentialsMode: Manual 

   1

   
   compute:
   - architecture: amd64
     hyperthreading: Enabled

   1

   This line is added to set the credentialsMode parameter to Manual.

3. Create the required OpenShift Container Platform installation manifests:

   $ openshift-install create manifests

4. Copy the manifests that

   ccoctl

   generated to the manifests directory that the installation program created:

   $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

5. Copy the private key that the

   ccoctl

   generated in the

   tls

   directory to the installation directory:

   $ cp -a /<path_to_ccoctl_output_dir>/tls .

6. Run the OpenShift Container Platform installer:

   $ ./openshift-install create cluster

1. Connect to the OpenShift Container Platform cluster.

2. Verify that the cluster does not have

   root

   credentials:

   $ oc get secrets -n kube-system aws-creds

   The output should look similar to:

   Error from server (NotFound): secrets "aws-creds" not found

3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

   Example command with the Image Registry Operator

   $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

   The output should show the role and web identity token that are used by the component and look similar to:

   Example output with the Image Registry Operator

   [default]
   role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
   web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

Procedure

1. Obtain the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Note

   Ensure that the architecture of the

   $RELEASE_IMAGE

   matches the architecture of the environment in which you will use the

   ccoctl

   tool.

3. Extract the

   ccoctl

   binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

4. Change the permissions to make

   ccoctl

   executable by running the following command:

   $ chmod 775 ccoctl

Procedure

1. Extract the list of

   CredentialsRequest

   objects from the OpenShift Container Platform release image by running the following command:

   $ oc adm release extract \
   --credentials-requests \
   --cloud=gcp \
   --to=<path_to_directory_with_list_of_credentials_requests>/credrequests \ 

   1

   
   quay.io/<path_to>/ocp-release:<version>

   1

   credrequests is the directory where the list of CredentialsRequest objects is stored. This command creates the directory if it does not exist.

   Note

   This command can take a few moments to run.

2. If your cluster uses cluster capabilities to disable one or more optional components, delete the

   CredentialsRequest

   custom resources for any disabled components.

   Example credrequests directory contents for OpenShift Container Platform 4.12 on GCP

   0000_26_cloud-controller-manager-operator_16_credentialsrequest-gcp.yaml 

   1

   
   0000_30_machine-api-operator_00_credentials-request.yaml 

   2

   
   0000_50_cloud-credential-operator_05-gcp-ro-credentialsrequest.yaml 

   3

   
   0000_50_cluster-image-registry-operator_01-registry-credentials-request-gcs.yaml 

   4

   
   0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml 

   5

   
   0000_50_cluster-network-operator_02-cncc-credentials.yaml 

   6

   
   0000_50_cluster-storage-operator_03_credentials_request_gcp.yaml 

   7

   1

   The Cloud Controller Manager Operator CR is required.

   2

   The Machine API Operator CR is required.

   3

   The Cloud Credential Operator CR is required.

   4

   The Image Registry Operator CR is required.

   5

   The Ingress Operator CR is required.

   6

   The Network Operator CR is required.

   7

   The Storage Operator CR is an optional component and might be disabled in your cluster.

3. Use the

   ccoctl

   tool to process all

   CredentialsRequest

   objects in the

   credrequests

   directory:

   $ ccoctl gcp create-all \
   --name=<name> \
   --region=<gcp_region> \
   --project=<gcp_project_id> \
   --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests

   where:

   • <name>

     is the user-defined name for all created GCP resources used for tracking.

   • <gcp_region>

     is the GCP region in which cloud resources will be created.

   • <gcp_project_id>

     is the GCP project ID in which cloud resources will be created.

   • <path_to_directory_with_list_of_credentials_requests>/credrequests

     is the directory containing the files of

     CredentialsRequest

     manifests to create GCP service accounts.
   Note

   If your cluster uses Technology Preview features that are enabled by the

   TechPreviewNoUpgrade

   feature set, you must include the

   --enable-tech-preview

   parameter.

Procedure

1. Change to the directory that contains the installation program and create the

   install-config.yaml

   file:

   $ openshift-install create install-config --dir <installation_directory>

   where

   <installation_directory>

   is the directory in which the installation program creates files.

2. Edit the

   install-config.yaml

   configuration file so that it contains the

   credentialsMode

   parameter set to

   Manual

   .

   Example install-config.yaml configuration file

   apiVersion: v1
   baseDomain: cluster1.example.com
   credentialsMode: Manual 

   1

   
   compute:
   - architecture: amd64
     hyperthreading: Enabled

   1

   This line is added to set the credentialsMode parameter to Manual.

3. Create the required OpenShift Container Platform installation manifests:

   $ openshift-install create manifests

4. Copy the manifests that

   ccoctl

   generated to the manifests directory that the installation program created:

   $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

5. Copy the private key that the

   ccoctl

   generated in the

   tls

   directory to the installation directory:

   $ cp -a /<path_to_ccoctl_output_dir>/tls .

6. Run the OpenShift Container Platform installer:

   $ ./openshift-install create cluster

1. Connect to the OpenShift Container Platform cluster.

2. Verify that the cluster does not have

   root

   credentials:

   $ oc get secrets -n kube-system gcp-credentials

   The output should look similar to:

   Error from server (NotFound): secrets "gcp-credentials" not found

3. Verify that the components are assuming the service accounts that are specified in the secret manifests, instead of using credentials that are created by the CCO:

   Example command with the Image Registry Operator

   $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r '.data."service_account.json"' | base64 -d

   The output should show the role and web identity token that are used by the component and look similar to:

   Example output with the Image Registry Operator

   {
      "type": "external_account", 

   1

   
      "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider",
      "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
      "token_url": "https://sts.googleapis.com/v1/token",
      "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client-email-address>:generateAccessToken", 

   2

   
      "credential_source": {
         "file": "/var/run/secrets/openshift/serviceaccount/token",
         "format": {
            "type": "text"
         }
      }
   }

   1

   The credential type is external_account.

   2

   The resource URL of the service account used by the Image Registry Operator.