Chapter 11. Changing the cloud provider credentials configuration

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   VMware vSphere	vsphere-creds

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command:

   $ oc get co kube-controller-manager

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \1
        -n <secret_namespace> 2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. Set a variable for the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE \
     --file="/usr/bin/ccoctl.<rhel_version>" \1
     -a ~/.pull-secret

   1

   For <rhel_version>, specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid:

   • rhel8: Specify this value for hosts that use RHEL 8.
   • rhel9: Specify this value for hosts that use RHEL 9.

4. Change the permissions to make ccoctl executable by running the following command:

   $ chmod 775 ccoctl.<rhel_version>

Procedure

1. Create an output directory for the manifests that the ccoctl utility generates. This procedure uses ./output_dir as an example.

2. Extract the service account public signing key for the cluster to the output directory by running the following command:

   $ oc get configmap \
     --namespace openshift-kube-apiserver bound-sa-token-signing-certs \
     --output 'go-template={{index .data "service-account-001.pub"}}' > ./output_dir/serviceaccount-signer.public 1

   1

   This procedure uses a file named serviceaccount-signer.public as an example.

3. Use the extracted service account public signing key to create an OpenID Connect (OIDC) issuer and Azure blob storage container with OIDC configuration files by running the following command:

   $ ./ccoctl azure create-oidc-issuer \
     --name <azure_infra_name> \1
     --output-dir ./output_dir \
     --region <azure_region> \2
     --subscription-id <azure_subscription_id> \3
     --tenant-id <azure_tenant_id> \
     --public-key-file ./output_dir/serviceaccount-signer.public 4

   1

   The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value.

   2

   Specify the region of the existing cluster.

   3

   Specify the subscription ID of the existing cluster.

   4

   Specify the file that contains the service account public signing key for the cluster.

4. Verify that the configuration file for the Azure pod identity webhook was created by running the following command:

   $ ll ./output_dir/manifests

   Example output

   total 8
   -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 1
   -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml

   1

   The file azure-ad-pod-identity-webhook-config.yaml contains the Azure pod identity webhook configuration.

5. Set an OIDC_ISSUER_URL variable with the OIDC issuer URL from the generated manifests in the output directory by running the following command:

   $ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`

6. Update the spec.serviceAccountIssuer parameter of the cluster authentication configuration by running the following command:

   $ oc patch authentication cluster \
     --type=merge \
     -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"

7. Monitor the configuration update progress by running the following command:

   $ oc adm wait-for-stable-cluster

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All clusteroperators are stable

8. Restart all of the pods in the cluster by running the following command:

   $ oc adm reboot-machine-config-pool mcp/worker mcp/master

   Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

9. Monitor the restart and update process by running the following command:

   $ oc adm wait-for-node-reboot nodes --all

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All nodes rebooted

10. Update the Cloud Credential Operator spec.credentialsMode parameter to Manual by running the following command:

    $ oc patch cloudcredential cluster \
      --type=merge \
      --patch '{"spec":{"credentialsMode":"Manual"}}'

11. Extract the list of CredentialsRequest objects from the OpenShift Container Platform release image by running the following command:

    $ oc adm release extract \
      --credentials-requests \
      --included \
      --to <path_to_directory_for_credentials_requests> \
      --registry-config ~/.pull-secret

    Note

    This command might take a few moments to run.

12. Set an AZURE_INSTALL_RG variable with the Azure resource group name by running the following command:

    $ AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`

13. Use the ccoctl utility to create managed identities for all CredentialsRequest objects by running the following command:

    $ ccoctl azure create-managed-identities \
      --name <azure_infra_name> \
      --output-dir ./output_dir \
      --region <azure_region> \
      --subscription-id <azure_subscription_id> \
      --credentials-requests-dir <path_to_directory_for_credentials_requests> \
      --issuer-url "${OIDC_ISSUER_URL}" \
      --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \1
      --installation-resource-group-name "${AZURE_INSTALL_RG}"

    1

    Specify the name of the resource group that contains the DNS zone.

14. Apply the Azure pod identity webhook configuration for Workload ID by running the following command:

    $ oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml

15. Apply the secrets generated by the ccoctl utility by running the following command:

    $ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}

    This process might take several minutes.

16. Restart all of the pods in the cluster by running the following command:

    $ oc adm reboot-machine-config-pool mcp/worker mcp/master

    Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

17. Monitor the restart and update process by running the following command:

    $ oc adm wait-for-node-reboot nodes --all

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All nodes rebooted

18. Monitor the configuration update progress by running the following command:

    $ oc adm wait-for-stable-cluster

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All clusteroperators are stable

19. Optional: Remove the Azure root credentials secret by running the following command:

    $ oc delete secret -n kube-system azure-credentials