Chapter 7. Preparing to update a cluster with manually maintained credentials
The Cloud Credential Operator (CCO) Upgradable
status for a cluster with manually maintained credentials is False
by default.
-
For minor releases, for example, from 4.12 to 4.13, this status prevents you from updating until you have addressed any updated permissions and annotated the
CloudCredential
resource to indicate that the permissions are updated as needed for the next version. This annotation changes theUpgradable
status toTrue
. - For z-stream releases, for example, from 4.13.0 to 4.13.1, no permissions are added or changed, so the update is not blocked.
Before updating a cluster with manually maintained credentials, you must accommodate any new or changed credentials in the release image for the version of OpenShift Container Platform you are updating to.
7.1. Update requirements for clusters with manually maintained credentials
Before you update a cluster that uses manually maintained credentials with the Cloud Credential Operator (CCO), you must update the cloud provider resources for the new release.
If the cloud credential management for your cluster was configured using the CCO utility (ccoctl
), use the ccoctl
utility to update the resources. Clusters that were configured to use manual mode without the ccoctl
utility require manual updates for the resources.
After updating the cloud provider resources, you must update the upgradeable-to
annotation for the cluster to indicate that it is ready to update.
The process to update the cloud provider resources and the upgradeable-to
annotation can only be completed by using command line tools.
7.1.1. Cloud credential configuration options and update requirements by platform type
Some platforms only support using the CCO in one mode. For clusters that are installed on those platforms, the platform type determines the credentials update requirements.
For platforms that support using the CCO in multiple modes, you must determine which mode the cluster is configured to use and take the required actions for that configuration.
Figure 7.1. Credentials update requirements by platform type
- Red Hat OpenStack Platform (RHOSP), Red Hat Virtualization (RHV), and VMware vSphere
These platforms do not support using the CCO in manual mode. Clusters on these platforms handle changes in cloud provider resources automatically and do not require an update to the
upgradeable-to
annotation.Administrators of clusters on these platforms should skip the manually maintained credentials section of the update process.
- IBM Cloud and Nutanix
Clusters installed on these platforms are configured using the
ccoctl
utility.Administrators of clusters on these platforms must take the following actions:
-
Configure the
ccoctl
utility for the new release. -
Use the
ccoctl
utility to update the cloud provider resources. -
Indicate that the cluster is ready to update with the
upgradeable-to
annotation.
-
Configure the
- Microsoft Azure Stack Hub
These clusters use manual mode with long-lived credentials and do not use the
ccoctl
utility.Administrators of clusters on these platforms must take the following actions:
- Manually update the cloud provider resources for the new release.
-
Indicate that the cluster is ready to update with the
upgradeable-to
annotation.
- Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP)
Clusters installed on these platforms support multiple CCO modes.
The required update process depends on the mode that the cluster is configured to use. If you are not sure what mode the CCO is configured to use on your cluster, you can use the web console or the CLI to determine this information.
Additional resources
- Determining the Cloud Credential Operator mode by using the web console
- Determining the Cloud Credential Operator mode by using the CLI
- Configuring the Cloud Credential Operator utility for a cluster update
- Updating cloud provider resources with manually maintained credentials
- About the Cloud Credential Operator
7.1.2. Determining the Cloud Credential Operator mode by using the web console
You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the web console.
Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.
Prerequisites
- You have access to an OpenShift Container Platform account with cluster administrator permissions.
Procedure
-
Log in to the OpenShift Container Platform web console as a user with the
cluster-admin
role. -
Navigate to Administration
Cluster Settings. - On the Cluster Settings page, select the Configuration tab.
- Under Configuration resource, select CloudCredential.
- On the CloudCredential details page, select the YAML tab.
In the YAML block, check the value of
spec.credentialsMode
. The following values are possible, though not all are supported on all platforms:-
''
: The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation. -
Mint
: The CCO is operating in mint mode. -
Passthrough
: The CCO is operating in passthrough mode. -
Manual
: The CCO is operating in manual mode.
ImportantTo determine the specific configuration of an AWS or GCP cluster that has a
spec.credentialsMode
of''
,Mint
, orManual
, you must investigate further.AWS and GCP clusters support using mint mode with the root secret deleted. If the cluster is specifically configured to use mint mode or uses mint mode by default, you must determine if the root secret is present on the cluster before updating.
An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster
Authentication
object.-
AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, navigate to Workloads
Secrets and look for the root secret for your cloud provider. NoteEnsure that the Project dropdown is set to All Projects.
Platform Secret name AWS
aws-creds
GCP
gcp-credentials
- If you see one of these values, your cluster is using mint or passthrough mode with the root secret present.
- If you do not see these values, your cluster is using the CCO in mint mode with the root secret removed.
AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, you must check the cluster
Authentication
object YAML values.-
Navigate to Administration
Cluster Settings. - On the Cluster Settings page, select the Configuration tab.
- Under Configuration resource, select Authentication.
- On the Authentication details page, select the YAML tab.
In the YAML block, check the value of the
.spec.serviceAccountIssuer
parameter.-
A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the
ccoctl
utility. -
An empty value (
''
) indicates that the cluster is using the CCO in manual mode but was not configured using theccoctl
utility.
-
A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the
-
Navigate to Administration
Next steps
- If you are updating a cluster that has the CCO operating in mint or passthrough mode and the root secret is present, you do not need to update any cloud provider resources and can continue to the next part of the update process.
- If your cluster is using the CCO in mint mode with the root secret removed, you must reinstate the credential secret with the administrator-level credential before continuing to the next part of the update process.
If your cluster was configured using the CCO utility (
ccoctl
), you must take the following actions:-
Configure the
ccoctl
utility for the new release and use it to update the cloud provider resources. -
Update the
upgradeable-to
annotation to indicate that the cluster is ready to update.
-
Configure the
If your cluster is using the CCO in manual mode but was not configured using the
ccoctl
utility, you must take the following actions:- Manually update the cloud provider resources for the new release.
-
Update the
upgradeable-to
annotation to indicate that the cluster is ready to update.
7.1.3. Determining the Cloud Credential Operator mode by using the CLI
You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the CLI.
Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.
Prerequisites
- You have access to an OpenShift Container Platform account with cluster administrator permissions.
-
You have installed the OpenShift CLI (
oc
).
Procedure
-
Log in to
oc
on the cluster as a user with thecluster-admin
role. To determine the mode that the CCO is configured to use, enter the following command:
$ oc get cloudcredentials cluster \ -o=jsonpath={.spec.credentialsMode}
The following output values are possible, though not all are supported on all platforms:
-
''
: The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation. -
Mint
: The CCO is operating in mint mode. -
Passthrough
: The CCO is operating in passthrough mode. -
Manual
: The CCO is operating in manual mode.
ImportantTo determine the specific configuration of an AWS or GCP cluster that has a
spec.credentialsMode
of''
,Mint
, orManual
, you must investigate further.AWS and GCP clusters support using mint mode with the root secret deleted. If the cluster is specifically configured to use mint mode or uses mint mode by default, you must determine if the root secret is present on the cluster before updating.
An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster
Authentication
object.-
AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, run the following command:
$ oc get secret <secret_name> \ -n=kube-system
where
<secret_name>
isaws-creds
for AWS orgcp-credentials
for GCP.If the root secret is present, the output of this command returns information about the secret. An error indicates that the root secret is not present on the cluster.
AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, run the following command:
$ oc get authentication cluster \ -o jsonpath \ --template='{ .spec.serviceAccountIssuer }'
This command displays the value of the
.spec.serviceAccountIssuer
parameter in the clusterAuthentication
object.-
An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the
ccoctl
utility. -
An empty output indicates that the cluster is using the CCO in manual mode but was not configured using the
ccoctl
utility.
-
An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the
Next steps
- If you are updating a cluster that has the CCO operating in mint or passthrough mode and the root secret is present, you do not need to update any cloud provider resources and can continue to the next part of the update process.
- If your cluster is using the CCO in mint mode with the root secret removed, you must reinstate the credential secret with the administrator-level credential before continuing to the next part of the update process.
If your cluster was configured using the CCO utility (
ccoctl
), you must take the following actions:-
Configure the
ccoctl
utility for the new release and use it to update the cloud provider resources. -
Update the
upgradeable-to
annotation to indicate that the cluster is ready to update.
-
Configure the
If your cluster is using the CCO in manual mode but was not configured using the
ccoctl
utility, you must take the following actions:- Manually update the cloud provider resources for the new release.
-
Update the
upgradeable-to
annotation to indicate that the cluster is ready to update.
7.2. Configuring the Cloud Credential Operator utility for a cluster update
To upgrade a cluster that uses the Cloud Credential Operator (CCO) in manual mode to create and manage cloud credentials from outside of the cluster, extract and prepare the CCO utility (ccoctl
) binary.
The ccoctl
utility is a Linux binary that must run in a Linux environment.
Prerequisites
- You have access to an OpenShift Container Platform account with cluster administrator access.
-
You have installed the OpenShift CLI (
oc
).
-
Your cluster was configured using the
ccoctl
utility to create and manage cloud credentials from outside of the cluster.
Procedure
Obtain the OpenShift Container Platform release image by running the following command:
$ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')
Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:
$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)
NoteEnsure that the architecture of the
$RELEASE_IMAGE
matches the architecture of the environment in which you will use theccoctl
tool.Extract the
ccoctl
binary from the CCO container image within the OpenShift Container Platform release image by running the following command:$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret
Change the permissions to make
ccoctl
executable by running the following command:$ chmod 775 ccoctl
Verification
To verify that
ccoctl
is ready to use, display the help file. Use a relative file name when you run the command, for example:$ ./ccoctl.rhel9
Example output
OpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: alibabacloud Manage credentials objects for alibaba cloud aws Manage credentials objects for AWS cloud gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for IBM Cloud nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command.
7.3. Updating cloud provider resources with the Cloud Credential Operator utility
The process for upgrading an OpenShift Container Platform cluster that was configured using the CCO utility (ccoctl
) is similar to creating the cloud provider resources during installation.
By default, ccoctl
creates objects in the directory in which the commands are run. To create the objects in a different directory, use the --output-dir
flag. This procedure uses <path_to_ccoctl_output_dir>
to refer to this directory.
On AWS clusters, some ccoctl
commands make AWS API calls to create or modify AWS resources. You can use the --dry-run
flag to avoid making API calls. Using this flag creates JSON files on the local file system instead. You can review and modify the JSON files and then apply them with the AWS CLI tool using the --cli-input-json
parameters.
Prerequisites
- Obtain the OpenShift Container Platform release image for the version that you are upgrading to.
-
Extract and prepare the
ccoctl
binary from the release image.
Procedure
Extract the list of
CredentialsRequest
custom resources (CRs) from the OpenShift Container Platform release image by running the following command:$ oc adm release extract --credentials-requests \ --cloud=<provider_type> \ --to=<path_to_directory_with_list_of_credentials_requests>/credrequests \ quay.io/<path_to>/ocp-release:<version>
where:
-
<provider_type>
is the value for your cloud provider. Valid values arealibabacloud
,aws
,gcp
,ibmcloud
, andnutanix
. -
credrequests
is the directory where the list ofCredentialsRequest
objects is stored. This command creates the directory if it does not exist.
-
For each
CredentialsRequest
CR in the release image, ensure that a namespace that matches the text in thespec.secretRef.namespace
field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.Sample AWS
CredentialsRequest
objectapiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: cloud-credential-operator-iam-ro namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - iam:GetUser - iam:GetUserPolicy - iam:ListAccessKeys resource: "*" secretRef: name: cloud-credential-operator-iam-ro-creds namespace: openshift-cloud-credential-operator 1
- 1
- This field indicates the namespace which needs to exist to hold the generated secret.
The
CredentialsRequest
CRs for other platforms have a similar format with different platform-specific values.For any
CredentialsRequest
CR for which the cluster does not already have a namespace with the name specified inspec.secretRef.namespace
, create the namespace by running the following command:$ oc create namespace <component_namespace>
Use the
ccoctl
tool to process allCredentialsRequest
objects in thecredrequests
directory by running the command for your cloud provider. The following commands processCredentialsRequest
objects:-
Alibaba Cloud:
ccoctl alibabacloud create-ram-users
-
Amazon Web Services (AWS):
ccoctl aws create-iam-roles
-
Google Cloud Platform (GCP):
ccoctl gcp create-all
-
IBM Cloud:
ccoctl ibmcloud create-service-id
-
Nutanix:
ccoctl nutanix create-shared-secrets
ImportantRefer to the
ccoctl
utility instructions in the installation content for your cloud provider for important platform-specific details about the required arguments and special considerations.For each
CredentialsRequest
object,ccoctl
creates the required provider resources and a permissions policy as defined in eachCredentialsRequest
object from the OpenShift Container Platform release image.-
Alibaba Cloud:
Apply the secrets to your cluster by running the following command:
$ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}
Verification
You can verify that the required provider resources and permissions policies are created by querying the cloud provider. For more information, refer to your cloud provider documentation on listing roles or service accounts.
Next steps
-
Update the
upgradeable-to
annotation to indicate that the cluster is ready to upgrade.
Additional resources
-
Creating Alibaba Cloud credentials for OpenShift Container Platform components with the
ccoctl
tool - Creating AWS resources with the Cloud Credential Operator utility
- Creating GCP resources with the Cloud Credential Operator utility
- Manually creating IAM for IBM Cloud VPC
- Configuring IAM for Nutanix
- Indicating that the cluster is ready to upgrade
7.4. Updating cloud provider resources with manually maintained credentials
Before upgrading a cluster with manually maintained credentials, you must create any new credentials for the release image that you are upgrading to. You must also review the required permissions for existing credentials and accommodate any new permissions requirements in the new release for those components.
Procedure
Extract and examine the
CredentialsRequest
custom resource for the new release.The "Manually creating IAM" section of the installation content for your cloud provider explains how to obtain and use the credentials required for your cloud.
Update the manually maintained credentials on your cluster:
-
Create new secrets for any
CredentialsRequest
custom resources that are added by the new release image. -
If the
CredentialsRequest
custom resources for any existing credentials that are stored in secrets have changed permissions requirements, update the permissions as required.
-
Create new secrets for any
If your cluster uses cluster capabilities to disable one or more optional components, delete the
CredentialsRequest
custom resources for any disabled components.Example
credrequests
directory contents for OpenShift Container Platform 4.12 on AWS0000_30_machine-api-operator_00_credentials-request.yaml 1 0000_50_cloud-credential-operator_05-iam-ro-credentialsrequest.yaml 2 0000_50_cluster-image-registry-operator_01-registry-credentials-request.yaml 3 0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml 4 0000_50_cluster-network-operator_02-cncc-credentials.yaml 5 0000_50_cluster-storage-operator_03_credentials_request_aws.yaml 6
- 1
- The Machine API Operator CR is required.
- 2
- The Cloud Credential Operator CR is required.
- 3
- The Image Registry Operator CR is required.
- 4
- The Ingress Operator CR is required.
- 5
- The Network Operator CR is required.
- 6
- The Storage Operator CR is an optional component and might be disabled in your cluster.
Example
credrequests
directory contents for OpenShift Container Platform 4.12 on GCP0000_26_cloud-controller-manager-operator_16_credentialsrequest-gcp.yaml 1 0000_30_machine-api-operator_00_credentials-request.yaml 2 0000_50_cloud-credential-operator_05-gcp-ro-credentialsrequest.yaml 3 0000_50_cluster-image-registry-operator_01-registry-credentials-request-gcs.yaml 4 0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml 5 0000_50_cluster-network-operator_02-cncc-credentials.yaml 6 0000_50_cluster-storage-operator_03_credentials_request_gcp.yaml 7
- 1
- The Cloud Controller Manager Operator CR is required.
- 2
- The Machine API Operator CR is required.
- 3
- The Cloud Credential Operator CR is required.
- 4
- The Image Registry Operator CR is required.
- 5
- The Ingress Operator CR is required.
- 6
- The Network Operator CR is required.
- 7
- The Storage Operator CR is an optional component and might be disabled in your cluster.
Next steps
-
Update the
upgradeable-to
annotation to indicate that the cluster is ready to upgrade.
7.5. Indicating that the cluster is ready to upgrade
The Cloud Credential Operator (CCO) Upgradable
status for a cluster with manually maintained credentials is False
by default.
Prerequisites
-
For the release image that you are upgrading to, you have processed any new credentials manually or by using the Cloud Credential Operator utility (
ccoctl
). -
You have installed the OpenShift CLI (
oc
).
Procedure
-
Log in to
oc
on the cluster as a user with thecluster-admin
role. Edit the
CloudCredential
resource to add anupgradeable-to
annotation within themetadata
field by running the following command:$ oc edit cloudcredential cluster
Text to add
... metadata: annotations: cloudcredential.openshift.io/upgradeable-to: <version_number> ...
Where
<version_number>
is the version that you are upgrading to, in the formatx.y.z
. For example, use4.12.2
for OpenShift Container Platform 4.12.2.It may take several minutes after adding the annotation for the upgradeable status to change.
Verification
-
In the Administrator perspective of the web console, navigate to Administration
Cluster Settings. To view the CCO status details, click cloud-credential in the Cluster Operators list.
-
If the Upgradeable status in the Conditions section is False, verify that the
upgradeable-to
annotation is free of typographical errors.
-
If the Upgradeable status in the Conditions section is False, verify that the
- When the Upgradeable status in the Conditions section is True, begin the OpenShift Container Platform upgrade.