Chapter 18. Managing cloud provider credentials

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. Delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \1
        -n <secret_namespace> 2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   RHV	ovirt-credentials
   VMware vSphere	vsphere-creds

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command:

   $ oc get co kube-controller-manager

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. Obtain the OpenShift Container Platform release image.

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

2. Get the CCO container image from the OpenShift Container Platform release image:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE)

   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image:

   $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

4. Change the permissions to make ccoctl executable:

   $ chmod 775 ccoctl

Procedure

1. Change to the directory that contains the installation program and create the install-config.yaml file:

   $ openshift-install create install-config --dir <installation_directory>

   where <installation_directory> is the directory in which the installation program creates files.

2. Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.

   Example install-config.yaml configuration file

   apiVersion: v1
   baseDomain: cluster1.example.com
   credentialsMode: Manual 1
   compute:
   - architecture: amd64
     hyperthreading: Enabled
   ...

   1

   This line is added to set the credentialsMode parameter to Manual.

3. Create the required OpenShift Container Platform installation manifests:

   $ openshift-install create manifests

4. Copy the manifests that ccoctl generated to the manifests directory that the installation program created:

   $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

5. Copy the private key that the ccoctl generated in the tls directory to the installation directory:

   $ cp -a /<path_to_ccoctl_output_dir>/tls .

6. Run the OpenShift Container Platform installer:

   $ ./openshift-install create cluster

1. Connect to the OpenShift Container Platform cluster.

2. Verify that the cluster does not have root credentials:

   $ oc get secrets -n kube-system aws-creds

   The output should look similar to:

   Error from server (NotFound): secrets "aws-creds" not found

3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

   Example command with the Image Registry Operator

   $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

   The output should show the role and web identity token that are used by the component and look similar to:

   Example output with the Image Registry Operator

   [default]
   role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
   web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

Procedure

1. Obtain the OpenShift Container Platform release image.

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

2. Get the CCO container image from the OpenShift Container Platform release image:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE)

   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image:

   $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

4. Change the permissions to make ccoctl executable:

   $ chmod 775 ccoctl

Procedure

1. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image:

   $ oc adm release extract \
   --credentials-requests \
   --cloud=aws \
   --to=<path_to_directory_with_list_of_credentials_requests>/credrequests \ 1
   --quay.io/<path_to>/ocp-release:<version>

   1

   credrequests is the directory where the list of CredentialsRequest objects is stored. This command creates the directory if it does not exist.

2. For each CredentialsRequest CR in the release image, ensure that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.

   Sample AWS CredentialsRequest object

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cloud-credential-operator-iam-ro
     namespace: openshift-cloud-credential-operator
   spec:
     secretRef:
       name: cloud-credential-operator-iam-ro-creds
       namespace: openshift-cloud-credential-operator 1
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - effect: Allow
         action:
         - iam:GetUser
         - iam:GetUserPolicy
         - iam:ListAccessKeys
         resource: "*"

   1

   This field indicates the namespace which needs to exist to hold the generated secret.

3. For any CredentialsRequest CR for which the cluster does not already have a namespace with the name specified in spec.secretRef.namespace, create the namespace:

   $ oc create namespace <component_namespace>

4. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

   $ ccoctl aws create-iam-roles \
   --name <name> \
   --region=<aws_region> \
   --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
   --identity-provider-arn arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

   where:

   • <name> is the name used to tag any cloud resources that are created for tracking. For upgrades, use the same value that was used for the initial installation.
   • <aws_account_id> is the AWS account ID.
   • <aws_region> is the AWS region in which cloud resources will be created.
   Note

   For AWS environments that use alternative IAM API endpoints, such as GovCloud, you must also specify your region with the --region parameter.

   For each CredentialsRequest object, ccoctl creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy as defined in each CredentialsRequest object from the OpenShift Container Platform release image.

5. Apply the secrets to your cluster:

   $ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

Procedure

1. Extract and examine the CredentialsRequest custom resource for the new release.

   The "Manually creating IAM" section of the installation content for your cloud provider explains how to obtain and use the credentials required for your cloud.

2. Update the manually maintained credentials on your cluster:

   • Create new secrets for any CredentialsRequest custom resources that are added by the new release image.
   • If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed their permissions requirements, update the permissions as required.

3. When all of the secrets are correct for the new release, indicate that the cluster is ready to upgrade:

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field:

      $ oc edit cloudcredential cluster

      Text to add

      ...
        metadata:
          annotations:
            cloudcredential.openshift.io/upgradeable-to: <version_number>
      ...

      Where <version_number> is the version you are upgrading to, in the format x.y.z. For example, 4.8.2 for OpenShift Container Platform 4.8.2.

      It may take several minutes after adding the annotation for the upgradeable status to change.

4. Verify that the CCO is upgradeable:

   1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.
   2. To view the CCO status details, click cloud-credential in the Cluster Operators list.
   3. If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.