Chapter 12. Changing the cloud provider credentials configuration
For supported configurations, you can change how OpenShift Container Platform authenticates with your cloud provider.
To determine which cloud credentials strategy your cluster uses, see Determining the Cloud Credential Operator mode.
12.1. Rotating or removing cloud provider credentials
After installing OpenShift Container Platform, some organizations require the rotation or removal of the cloud provider credentials that were used during the initial installation.
To allow the cluster to use the new credentials, you must update the secrets that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.
12.1.1. Rotating cloud provider credentials with the Cloud Credential Operator utility
The Cloud Credential Operator (CCO) utility ccoctl
supports updating secrets for clusters installed on IBM Cloud®.
12.1.1.1. Rotating API keys
You can rotate API keys for your existing service IDs and update the corresponding secrets.
Prerequisites
-
You have configured the
ccoctl
binary. - You have existing service IDs in a live OpenShift Container Platform cluster installed.
Procedure
Use the
ccoctl
utility to rotate your API keys for the service IDs and update the secrets:$ ccoctl <provider_name> refresh-keys \ 1 --kubeconfig <openshift_kubeconfig_file> \ 2 --credentials-requests-dir <path_to_credential_requests_directory> \ 3 --name <name> 4
NoteIf your cluster uses Technology Preview features that are enabled by the
TechPreviewNoUpgrade
feature set, you must include the--enable-tech-preview
parameter.
12.1.2. Maintaining cloud provider credentials
If your cloud provider credentials are changed for any reason, you must manually update the secret that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.
The process for rotating cloud credentials depends on the mode that the CCO is configured to use. After you rotate credentials for a cluster that is using mint mode, you must manually remove the component credentials that were created by the removed credential.
Prerequisites
Your cluster is installed on a platform that supports rotating cloud credentials manually with the CCO mode that you are using:
- For mint mode, Amazon Web Services (AWS) and Google Cloud Platform (GCP) are supported.
- For passthrough mode, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Red Hat OpenStack Platform (RHOSP), and VMware vSphere are supported.
- You have changed the credentials that are used to interface with your cloud provider.
- The new credentials have sufficient permissions for the mode CCO is configured to use in your cluster.
Procedure
-
In the Administrator perspective of the web console, navigate to Workloads
Secrets. In the table on the Secrets page, find the root secret for your cloud provider.
Platform Secret name AWS
aws-creds
Azure
azure-credentials
GCP
gcp-credentials
RHOSP
openstack-credentials
VMware vSphere
vsphere-creds
- Click the Options menu in the same row as the secret and select Edit Secret.
- Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
- Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.
If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.
NoteIf the vSphere CSI Driver Operator is enabled, this step is not required.
To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the
cluster-admin
role and run the following command:$ oc patch kubecontrollermanager cluster \ -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \ --type=merge
While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports
Progressing=true
. To view the status, run the following command:$ oc get co kube-controller-manager
-
Log in to the OpenShift Container Platform CLI as a user with the
cluster-admin
role. Get the names and namespaces of all referenced component secrets:
$ oc -n openshift-cloud-credential-operator get CredentialsRequest \ -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'
where
<provider_spec>
is the corresponding value for your cloud provider:-
AWS:
AWSProviderSpec
-
GCP:
GCPProviderSpec
Partial example output for AWS
{ "name": "ebs-cloud-credentials", "namespace": "openshift-cluster-csi-drivers" } { "name": "cloud-credential-operator-iam-ro-creds", "namespace": "openshift-cloud-credential-operator" }
-
AWS:
Delete each of the referenced component secrets:
$ oc delete secret <secret_name> \1 -n <secret_namespace> 2
Example deletion of an AWS secret
$ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers
You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.
-
Log in to the OpenShift Container Platform CLI as a user with the
Verification
To verify that the credentials have changed:
-
In the Administrator perspective of the web console, navigate to Workloads
Secrets. - Verify that the contents of the Value field or fields have changed.
Additional resources
12.1.3. Removing cloud provider credentials
After installing an OpenShift Container Platform cluster with the Cloud Credential Operator (CCO) in mint mode, you can remove the administrator-level credential secret from the kube-system
namespace in the cluster. The administrator-level credential is required only during changes that require its elevated permissions, such as upgrades.
Prior to a non z-stream upgrade, you must reinstate the credential secret with the administrator-level credential. If the credential is not present, the upgrade might be blocked.
Prerequisites
- Your cluster is installed on a platform that supports removing cloud credentials from the CCO. Supported platforms are AWS and GCP.
Procedure
-
In the Administrator perspective of the web console, navigate to Workloads
Secrets. In the table on the Secrets page, find the root secret for your cloud provider.
Platform Secret name AWS
aws-creds
GCP
gcp-credentials
- Click the Options menu in the same row as the secret and select Delete Secret.
Additional resources
12.2. Enabling token-based authentication
After installing an Microsoft Azure OpenShift Container Platform cluster, you can enable Microsoft Entra Workload ID to use short-term credentials.
12.2.1. Configuring the Cloud Credential Operator utility
To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode, extract and prepare the CCO utility (ccoctl
) binary.
The ccoctl
utility is a Linux binary that must run in a Linux environment.
Prerequisites
- You have access to an OpenShift Container Platform account with cluster administrator access.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Set a variable for the OpenShift Container Platform release image by running the following command:
$ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')
Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:
$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)
NoteEnsure that the architecture of the
$RELEASE_IMAGE
matches the architecture of the environment in which you will use theccoctl
tool.Extract the
ccoctl
binary from the CCO container image within the OpenShift Container Platform release image by running the following command:$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret
Change the permissions to make
ccoctl
executable by running the following command:$ chmod 775 ccoctl
Verification
To verify that
ccoctl
is ready to use, display the help file. Use a relative file name when you run the command, for example:$ ./ccoctl.rhel9
Example output
OpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: alibabacloud Manage credentials objects for alibaba cloud aws Manage credentials objects for AWS cloud azure Manage credentials objects for Azure gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for IBM Cloud nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command.
12.2.2. Enabling Microsoft Entra Workload ID on an existing cluster
If you did not configure your Microsoft Azure OpenShift Container Platform cluster to use Microsoft Entra Workload ID during installation, you can enable this authentication method on an existing cluster.
The process to enable Workload ID on an existing cluster is disruptive and takes a significant amount of time. Before proceeding, observe the following considerations:
- Read the following steps and ensure that you understand and accept the time requirement. The exact time requirement varies depending on the individual cluster, but it is likely to require at least one hour.
- During this process, you must refresh all service accounts and restart all pods on the cluster. These actions are disruptive to workloads. To mitigate this impact, you can temporarily halt these services and then redeploy them when the cluster is ready.
- After starting this process, do not attempt to update the cluster until it is complete. If an update is triggered, the process to enable Workload ID on an existing cluster fails.
Prerequisites
- You have installed an OpenShift Container Platform cluster on Microsoft Azure.
-
You have access to the cluster using an account with
cluster-admin
permissions. -
You have installed the OpenShift CLI (
oc
). -
You have extracted and prepared the Cloud Credential Operator utility (
ccoctl
) binary. -
You have access to your Azure account by using the Azure CLI (
az
).
Procedure
-
Create an output directory for the manifests that the
ccoctl
utility generates. This procedure uses./output_dir
as an example. Extract the service account public signing key for the cluster to the output directory by running the following command:
$ oc get configmap \ --namespace openshift-kube-apiserver bound-sa-token-signing-certs \ --output 'go-template={{index .data "service-account-001.pub"}}' > ./output_dir/serviceaccount-signer.public 1
- 1
- This procedure uses a file named
serviceaccount-signer.public
as an example.
Use the extracted service account public signing key to create an OpenID Connect (OIDC) issuer and Azure blob storage container with OIDC configuration files by running the following command:
$ ./ccoctl azure create-oidc-issuer \ --name <azure_infra_name> \1 --output-dir ./output_dir \ --region <azure_region> \2 --subscription-id <azure_subscription_id> \3 --tenant-id <azure_tenant_id> \ --public-key-file ./output_dir/serviceaccount-signer.public 4
- 1
- The value of the
name
parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the--oidc-resource-group-name
argument with the existing group name as its value. - 2
- Specify the region of the existing cluster.
- 3
- Specify the subscription ID of the existing cluster.
- 4
- Specify the file that contains the service account public signing key for the cluster.
Verify that the configuration file for the Azure pod identity webhook was created by running the following command:
$ ll ./output_dir/manifests
Example output
total 8 -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 1 -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml
- 1
- The file
azure-ad-pod-identity-webhook-config.yaml
contains the Azure pod identity webhook configuration.
Set an
OIDC_ISSUER_URL
variable with the OIDC issuer URL from the generated manifests in the output directory by running the following command:$ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`
Update the
spec.serviceAccountIssuer
parameter of the clusterauthentication
configuration by running the following command:$ oc patch authentication cluster \ --type=merge \ -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"
Monitor the configuration update progress by running the following command:
$ oc adm wait-for-stable-cluster
This process might take 15 minutes or longer. The following output indicates that the process is complete:
All clusteroperators are stable
Restart all of the pods in the cluster by running the following command:
$ oc adm reboot-machine-config-pool mcp/worker mcp/master
Restarting a pod updates the
serviceAccountIssuer
field and refreshes the service account public signing key.Monitor the restart and update process by running the following command:
$ oc adm wait-for-node-reboot nodes --all
This process might take 15 minutes or longer. The following output indicates that the process is complete:
All nodes rebooted
Update the Cloud Credential Operator
spec.credentialsMode
parameter toManual
by running the following command:$ oc patch cloudcredential cluster \ --type=merge \ --patch '{"spec":{"credentialsMode":"Manual"}}'
Extract the list of
CredentialsRequest
objects from the OpenShift Container Platform release image by running the following command:$ oc adm release extract \ --credentials-requests \ --included \ --to <path_to_directory_for_credentials_requests> \ --registry-config ~/.pull-secret
NoteThis command might take a few moments to run.
Set an
AZURE_INSTALL_RG
variable with the Azure resource group name by running the following command:$ AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`
Use the
ccoctl
utility to create managed identities for allCredentialsRequest
objects by running the following command:$ ccoctl azure create-managed-identities \ --name <azure_infra_name> \ --output-dir ./output_dir \ --region <azure_region> \ --subscription-id <azure_subscription_id> \ --credentials-requests-dir <path_to_directory_for_credentials_requests> \ --issuer-url "${OIDC_ISSUER_URL}" \ --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \1 --installation-resource-group-name "${AZURE_INSTALL_RG}"
- 1
- Specify the name of the resource group that contains the DNS zone.
Apply the Azure pod identity webhook configuration for Workload ID by running the following command:
$ oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml
Apply the secrets generated by the
ccoctl
utility by running the following command:$ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}
This process might take several minutes.
Restart all of the pods in the cluster by running the following command:
$ oc adm reboot-machine-config-pool mcp/worker mcp/master
Restarting a pod updates the
serviceAccountIssuer
field and refreshes the service account public signing key.Monitor the restart and update process by running the following command:
$ oc adm wait-for-node-reboot nodes --all
This process might take 15 minutes or longer. The following output indicates that the process is complete:
All nodes rebooted
Monitor the configuration update progress by running the following command:
$ oc adm wait-for-stable-cluster
This process might take 15 minutes or longer. The following output indicates that the process is complete:
All clusteroperators are stable
Optional: Remove the Azure root credentials secret by running the following command:
$ oc delete secret -n kube-system azure-credentials
Additional resources
12.2.3. Verifying that a cluster uses short-term credentials
You can verify that a cluster uses short-term security credentials for individual components by checking the Cloud Credential Operator (CCO) configuration and other values in the cluster.
Prerequisites
-
You deployed an OpenShift Container Platform cluster using the Cloud Credential Operator utility (
ccoctl
) to implement short-term credentials. -
You installed the OpenShift CLI (
oc
). -
You are logged in as a user with
cluster-admin
privileges.
Procedure
Verify that the CCO is configured to operate in manual mode by running the following command:
$ oc get cloudcredentials cluster \ -o=jsonpath={.spec.credentialsMode}
The following output confirms that the CCO is operating in manual mode:
Example output
Manual
Verify that the cluster does not have
root
credentials by running the following command:$ oc get secrets \ -n kube-system <secret_name>
where
<secret_name>
is the name of the root secret for your cloud provider.Platform Secret name Amazon Web Services (AWS)
aws-creds
Microsoft Azure
azure-credentials
Google Cloud Platform (GCP)
gcp-credentials
An error confirms that the root secret is not present on the cluster.
Example output for an AWS cluster
Error from server (NotFound): secrets "aws-creds" not found
Verify that the components are using short-term security credentials for individual components by running the following command:
$ oc get authentication cluster \ -o jsonpath \ --template='{ .spec.serviceAccountIssuer }'
This command displays the value of the
.spec.serviceAccountIssuer
parameter in the clusterAuthentication
object. An output of a URL that is associated with your cloud provider indicates that the cluster is using manual mode with short-term credentials that are created and managed from outside of the cluster.Azure clusters: Verify that the components are assuming the Azure client ID that is specified in the secret manifests by running the following command:
$ oc get secrets \ -n openshift-image-registry installer-cloud-credentials \ -o jsonpath='{.data}'
An output that contains the
azure_client_id
andazure_federated_token_file
felids confirms that the components are assuming the Azure client ID.Azure clusters: Verify that the pod identity webhook is running by running the following command:
$ oc get pods \ -n openshift-cloud-credential-operator
Example output
NAME READY STATUS RESTARTS AGE cloud-credential-operator-59cf744f78-r8pbq 2/2 Running 2 71m pod-identity-webhook-548f977b4c-859lz 1/1 Running 1 70m