Chapter 10. Changing the cloud provider credentials configuration

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   VMware vSphere	vsphere-creds

   Show more
3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   Copy to Clipboard Toggle word wrap

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command:

   $ oc get co kube-controller-manager

   Copy to Clipboard Toggle word wrap

7. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      Copy to Clipboard Toggle word wrap

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

      Copy to Clipboard Toggle word wrap

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \

      1

      
        -n <secret_namespace> 

      2

      Copy to Clipboard Toggle word wrap

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      Copy to Clipboard Toggle word wrap

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

   Show more
3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. Set a variable for the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(oc get clusterversion -o jsonpath={..desired.image})

   Copy to Clipboard Toggle word wrap

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Copy to Clipboard Toggle word wrap
   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE \
     --file="/usr/bin/ccoctl.<rhel_version>" \

   1

   
     -a ~/.pull-secret

   Copy to Clipboard Toggle word wrap

   1

   For <rhel_version>, specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid:

   • rhel8: Specify this value for hosts that use RHEL 8.
   • rhel9: Specify this value for hosts that use RHEL 9.

4. Change the permissions to make ccoctl executable by running the following command:

   $ chmod 775 ccoctl.<rhel_version>

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an output directory for the manifests that the ccoctl utility generates. This procedure uses ./output_dir as an example.

2. Extract the service account public signing key for the cluster to the output directory by running the following command:

   $ oc get configmap \
     --namespace openshift-kube-apiserver bound-sa-token-signing-certs \
     --output 'go-template={{index .data "service-account-001.pub"}}' > ./output_dir/serviceaccount-signer.public 

   1

   Copy to Clipboard Toggle word wrap

   1

   This procedure uses a file named serviceaccount-signer.public as an example.

3. Use the extracted service account public signing key to create an OpenID Connect (OIDC) issuer and Azure blob storage container with OIDC configuration files by running the following command:

   $ ./ccoctl azure create-oidc-issuer \
     --name <azure_infra_name> \

   1

   
     --output-dir ./output_dir \
     --region <azure_region> \

   2

   
     --subscription-id <azure_subscription_id> \

   3

   
     --tenant-id <azure_tenant_id> \
     --public-key-file ./output_dir/serviceaccount-signer.public 

   4

   Copy to Clipboard Toggle word wrap

   1

   The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value.

   2

   Specify the region of the existing cluster.

   3

   Specify the subscription ID of the existing cluster.

   4

   Specify the file that contains the service account public signing key for the cluster.

4. Verify that the configuration file for the Azure pod identity webhook was created by running the following command:

   $ ll ./output_dir/manifests

   Copy to Clipboard Toggle word wrap

   Example output

   total 8
   -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 

   1

   
   -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml

   Copy to Clipboard Toggle word wrap

   1

   The file azure-ad-pod-identity-webhook-config.yaml contains the Azure pod identity webhook configuration.

5. Set an OIDC_ISSUER_URL variable with the OIDC issuer URL from the generated manifests in the output directory by running the following command:

   $ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`

   Copy to Clipboard Toggle word wrap

6. Update the spec.serviceAccountIssuer parameter of the cluster authentication configuration by running the following command:

   $ oc patch authentication cluster \
     --type=merge \
     -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"

   Copy to Clipboard Toggle word wrap

7. Monitor the configuration update progress by running the following command:

   $ oc adm wait-for-stable-cluster

   Copy to Clipboard Toggle word wrap

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All clusteroperators are stable

   Copy to Clipboard Toggle word wrap

8. Restart all of the pods in the cluster by running the following command:

   $ oc adm reboot-machine-config-pool mcp/worker mcp/master

   Copy to Clipboard Toggle word wrap

   Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

9. Monitor the restart and update process by running the following command:

   $ oc adm wait-for-node-reboot nodes --all

   Copy to Clipboard Toggle word wrap

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All nodes rebooted

   Copy to Clipboard Toggle word wrap

10. Update the Cloud Credential Operator spec.credentialsMode parameter to Manual by running the following command:

    $ oc patch cloudcredential cluster \
      --type=merge \
      --patch '{"spec":{"credentialsMode":"Manual"}}'

    Copy to Clipboard Toggle word wrap

11. Extract the list of CredentialsRequest objects from the OpenShift Container Platform release image by running the following command:

    $ oc adm release extract \
      --credentials-requests \
      --included \
      --to <path_to_directory_for_credentials_requests> \
      --registry-config ~/.pull-secret

    Copy to Clipboard Toggle word wrap
    Note

    This command might take a few moments to run.

12. Set an AZURE_INSTALL_RG variable with the Azure resource group name by running the following command:

    $ AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`

    Copy to Clipboard Toggle word wrap

13. Use the ccoctl utility to create managed identities for all CredentialsRequest objects by running the following command:

    Note

    The following command does not show all available options. For a complete list of options, including those that might be necessary for your specific use case, run $ ccoctl azure create-managed-identities --help.

    $ ccoctl azure create-managed-identities \
      --name <azure_infra_name> \
      --output-dir ./output_dir \
      --region <azure_region> \
      --subscription-id <azure_subscription_id> \
      --credentials-requests-dir <path_to_directory_for_credentials_requests> \
      --issuer-url "${OIDC_ISSUER_URL}" \
      --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \

    1

    
      --installation-resource-group-name "${AZURE_INSTALL_RG}" \
      --network-resource-group-name <azure_resource_group> 

    2

    Copy to Clipboard Toggle word wrap

    1

    Specify the name of the resource group that contains the DNS zone.

    2

    Optional: Specify the virtual network resource group if it is different from the cluster resource group.

14. Apply the Azure pod identity webhook configuration for Workload ID by running the following command:

    $ oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml

    Copy to Clipboard Toggle word wrap

15. Apply the secrets generated by the ccoctl utility by running the following command:

    $ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}

    Copy to Clipboard Toggle word wrap

    This process might take several minutes.

16. Restart all of the pods in the cluster by running the following command:

    $ oc adm reboot-machine-config-pool mcp/worker mcp/master

    Copy to Clipboard Toggle word wrap

    Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

17. Monitor the restart and update process by running the following command:

    $ oc adm wait-for-node-reboot nodes --all

    Copy to Clipboard Toggle word wrap

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All nodes rebooted

    Copy to Clipboard Toggle word wrap

18. Monitor the configuration update progress by running the following command:

    $ oc adm wait-for-stable-cluster

    Copy to Clipboard Toggle word wrap

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All clusteroperators are stable

    Copy to Clipboard Toggle word wrap

19. Optional: Remove the Azure root credentials secret by running the following command:

    $ oc delete secret -n kube-system azure-credentials

    Copy to Clipboard Toggle word wrap