Chapter 2. Preparing to update a cluster

Procedure

1. List all the Gateway API CRDs that you need to remove by running the following command:

   $ oc get crd | grep -F -e gateway.networking.k8s.io -e gateway.networking.x-k8s.io

   Copy to Clipboard

   Example output

   gatewayclasses.gateway.networking.k8s.io
   gateways.gateway.networking.k8s.io
   grpcroutes.gateway.networking.k8s.io
   httproutes.gateway.networking.k8s.io
   referencegrants.gateway.networking.k8s.io

   Copy to Clipboard

2. Delete the Gateway API CRDs from the previous step by running the following command:

   $ oc delete crd gatewayclasses.networking.k8s.io && \
   oc delete crd gateways.networking.k8s.io && \
   oc delete crd grpcroutes.gateway.networking.k8s.io && \
   oc delete crd httproutes.gateway.networking.k8s.io && \
   oc delete crd referencesgrants.gateway.networking.k8s.io

   Copy to Clipboard
   Important

   Deleting CRDs removes every custom resource that relies on them and can result in data loss. Back up any necessary data before deleting the Gateway API CRDs. Any controller that was previously managing the lifecycle of the Gateway API CRDs will fail to operate properly. Attempting to force its use in conjunction with the Ingress Operator to manage Gateway API CRDs might prevent the cluster update from succeeding.

3. Get the supported Gateway API CRDs by running the following command:

   $ oc apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

   Copy to Clipboard
   Warning

   You can perform this step without deleting your CRDs. If your update to a CRD removes a field that is used by a custom resource, you can lose data. Updating a CRD a second time, to a version that re-adds a field, can cause any previously deleted data to reappear. Any third-party controller that depends on a specific Gateway API CRD version that is not supported in OpenShift Container Platform 4.19 will break upon updating that CRD to one supported by Red Hat.

   For more information on the OpenShift Container Platform implementation and the dead fields issue, see Gateway API implementation for OpenShift Container Platform.

Procedure

1. Obtain the pull spec for the update that you want to apply by running the following command:

   $ oc adm upgrade

   Copy to Clipboard

   The output of this command includes pull specs for the available updates similar to the following:

   Partial example output

   ...
   Recommended updates:
   
   VERSION IMAGE
   4.19.0  quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
   ...

   Copy to Clipboard

2. Set a $RELEASE_IMAGE variable with the release image that you want to use by running the following command:

   $ RELEASE_IMAGE=<update_pull_spec>

   Copy to Clipboard

   where <update_pull_spec> is the pull spec for the release image that you want to use. For example:

   quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032

   Copy to Clipboard

3. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image by running the following command:

   $ oc adm release extract \
     --from=$RELEASE_IMAGE \
     --credentials-requests \
     --included \

   1

   
     --to=<path_to_directory_for_credentials_requests> 

   2

   Copy to Clipboard

   1

   The --included parameter includes only the manifests that your specific cluster configuration requires for the target release.

   2

   Specify the path to the directory where you want to store the CredentialsRequest objects. If the specified directory does not exist, this command creates it.

   This command creates a YAML file for each CredentialsRequest object.

4. For each CredentialsRequest CR in the release image, ensure that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.

   Sample AWS CredentialsRequest object

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cloud-credential-operator-iam-ro
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - effect: Allow
         action:
         - iam:GetUser
         - iam:GetUserPolicy
         - iam:ListAccessKeys
         resource: "*"
     secretRef:
       name: cloud-credential-operator-iam-ro-creds
       namespace: openshift-cloud-credential-operator 

   1

   Copy to Clipboard

   1

   This field indicates the namespace which must exist to hold the generated secret.

   The CredentialsRequest CRs for other platforms have a similar format with different platform-specific values.

5. For any CredentialsRequest CR for which the cluster does not already have a namespace with the name specified in spec.secretRef.namespace, create the namespace by running the following command:

   $ oc create namespace <component_namespace>

   Copy to Clipboard

Procedure

1. Set a variable for the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(oc get clusterversion -o jsonpath={..desired.image})

   Copy to Clipboard

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Copy to Clipboard
   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE \
     --file="/usr/bin/ccoctl.<rhel_version>" \

   1

   
     -a ~/.pull-secret

   Copy to Clipboard

   1

   For <rhel_version>, specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid:

   • rhel8: Specify this value for hosts that use RHEL 8.
   • rhel9: Specify this value for hosts that use RHEL 9.

4. Change the permissions to make ccoctl executable by running the following command:

   $ chmod 775 ccoctl.<rhel_version>

   Copy to Clipboard

Procedure

1. Use the ccoctl tool to process all CredentialsRequest objects by running the command for your cloud provider. The following commands process CredentialsRequest objects:

   Example 2.1. Amazon Web Services (AWS)

   $ ccoctl aws create-all \

   1

   
     --name=<name> \

   2

   
     --region=<aws_region> \

   3

   
     --credentials-requests-dir=<path_to_credentials_requests_directory> \

   4

   
     --output-dir=<path_to_ccoctl_output_dir> \

   5

   
     --create-private-s3-bucket 

   6

   Copy to Clipboard

   1

   To create the AWS resources individually, use the "Creating AWS resources individually" procedure in the "Installing a cluster on AWS with customizations" content. This option might be useful if you need to review the JSON files that the ccoctl tool creates before modifying AWS resources, or if the process the ccoctl tool uses to create AWS resources automatically does not meet the requirements of your organization.

   2

   Specify the name used to tag any cloud resources that are created for tracking.

   3

   Specify the AWS region in which cloud resources will be created.

   4

   Specify the directory containing the files for the component CredentialsRequest objects.

   5

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   6

   Optional: By default, the ccoctl utility stores the OpenID Connect (OIDC) configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint. To store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL instead, use the --create-private-s3-bucket parameter.

   Example 2.2. Google Cloud Platform (GCP)

   $ ccoctl gcp create-all \
     --name=<name> \

   1

   
     --region=<gcp_region> \

   2

   
     --project=<gcp_project_id> \

   3

   
     --credentials-requests-dir=<path_to_credentials_requests_directory> \

   4

   
     --output-dir=<path_to_ccoctl_output_dir> 

   5

   Copy to Clipboard

   1

   Specify the user-defined name for all created GCP resources used for tracking.

   2

   Specify the GCP region in which cloud resources will be created.

   3

   Specify the GCP project ID in which cloud resources will be created.

   4

   Specify the directory containing the files of CredentialsRequest manifests to create GCP service accounts.

   5

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   Example 2.3. IBM Cloud

   $ ccoctl ibmcloud create-service-id \
     --credentials-requests-dir=<path_to_credential_requests_directory> \

   1

   
     --name=<cluster_name> \

   2

   
     --output-dir=<installation_directory> \

   3

   
     --resource-group-name=<resource_group_name> 

   4

   Copy to Clipboard

   1

   Specify the directory containing the files for the component CredentialsRequest objects.

   2

   Specify the name of the OpenShift Container Platform cluster.

   3

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   4

   Optional: Specify the name of the resource group used for scoping the access policies.

   Example 2.4. Microsoft Azure

   $ ccoctl azure create-managed-identities \
     --name <azure_infra_name> \

   1

   
     --output-dir ./output_dir \
     --region <azure_region> \

   2

   
     --subscription-id <azure_subscription_id> \

   3

   
     --credentials-requests-dir <path_to_directory_for_credentials_requests> \
     --issuer-url "${OIDC_ISSUER_URL}" \

   4

   
     --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \

   5

   
     --installation-resource-group-name "${AZURE_INSTALL_RG}" 

   6

   Copy to Clipboard

   1

   The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value.

   2

   Specify the region of the existing cluster.

   3

   Specify the subscription ID of the existing cluster.

   4

   Specify the OIDC issuer URL from the existing cluster. You can obtain this value by running the following command:

   $ oc get authentication cluster \
     -o jsonpath \
     --template='{ .spec.serviceAccountIssuer }'

   Copy to Clipboard

   5

   Specify the name of the resource group that contains the DNS zone.

   6

   Specify the Azure resource group name. You can obtain this value by running the following command:

   $ oc get infrastructure cluster \
     -o jsonpath \
     --template '{ .status.platformStatus.azure.resourceGroupName }'

   Copy to Clipboard

   Example 2.5. Nutanix

   $ ccoctl nutanix create-shared-secrets \
     --credentials-requests-dir=<path_to_credentials_requests_directory> \

   1

   
     --output-dir=<ccoctl_output_dir> \

   2

   
     --credentials-source-filepath=<path_to_credentials_file> 

   3

   Copy to Clipboard

   1

   Specify the path to the directory that contains the files for the component CredentialsRequests objects.

   2

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   3

   Optional: Specify the directory that contains the credentials data YAML file. By default, ccoctl expects this file to be in <home_directory>/.nutanix/credentials.

   For each CredentialsRequest object, ccoctl creates the required provider resources and a permissions policy as defined in each CredentialsRequest object from the OpenShift Container Platform release image.

2. Apply the secrets to your cluster by running the following command:

   $ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

   Copy to Clipboard

Procedure

1. Create YAML files with secrets for any CredentialsRequest custom resources that the new release image adds. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each CredentialsRequest object.

   Example 2.6. Sample AWS YAML files

   Sample AWS CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - effect: Allow
         action:
         - s3:CreateBucket
         - s3:DeleteBucket
         resource: "*"
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Copy to Clipboard

   Sample AWS Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     aws_access_key_id: <base64_encoded_aws_access_key_id>
     aws_secret_access_key: <base64_encoded_aws_secret_access_key>

   Copy to Clipboard

   Example 2.7. Sample Azure YAML files

   Note

   Global Azure and Azure Stack Hub use the same CredentialsRequest object and secret formats.

   Sample Azure CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Copy to Clipboard

   Sample Azure Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     azure_subscription_id: <base64_encoded_azure_subscription_id>
     azure_client_id: <base64_encoded_azure_client_id>
     azure_client_secret: <base64_encoded_azure_client_secret>
     azure_tenant_id: <base64_encoded_azure_tenant_id>
     azure_resource_prefix: <base64_encoded_azure_resource_prefix>
     azure_resourcegroup: <base64_encoded_azure_resourcegroup>
     azure_region: <base64_encoded_azure_region>

   Copy to Clipboard

   Example 2.8. Sample GCP YAML files

   Sample GCP CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: GCPProviderSpec
         predefinedRoles:
         - roles/iam.securityReviewer
         - roles/iam.roleViewer
         skipServiceCheck: true
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Copy to Clipboard

   Sample GCP Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     service_account.json: <base64_encoded_gcp_service_account_file>

   Copy to Clipboard

2. If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed permissions requirements, update the permissions as required.

Procedure

1. Log in to oc on the cluster as a user with the cluster-admin role.

2. Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field by running the following command:

   $ oc edit cloudcredential cluster

   Copy to Clipboard

   Text to add

   ...
     metadata:
       annotations:
         cloudcredential.openshift.io/upgradeable-to: <version_number>
   ...

   Copy to Clipboard

   Where <version_number> is the version that you are upgrading to, in the format x.y.z. For example, use 4.12.2 for OpenShift Container Platform 4.12.2.

   It may take several minutes after adding the annotation for the upgradeable status to change.

Verification

1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.

2. To view the CCO status details, click cloud-credential in the Cluster Operators list.

   • If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.
3. When the Upgradeable status in the Conditions section is True, begin the OpenShift Container Platform upgrade.

Procedure

1. Deschedule the RHEL node currently running your existing pods by entering the following command:

   $ oc adm cordon <rhel-node>

   Copy to Clipboard

2. Add an rhcos node selector to a pod:

   • To add the node selector to existing and future pods, add the node selector to the controller object for the pods by entering the following command:

     Example Deployment object with rhcos label

     $ oc patch dc <my-app> -p '{"spec":{"template":{"spec":{"nodeSelector":{"node.openshift.io/os_id":"rhcos"}}}}}'

     Copy to Clipboard

     Any existing pods under your Deployment controlling object will be re-created on your RHCOS nodes.

   • To add the node selector to a specific, new pod, add the selector to the Pod object directly:

     Example Pod object with rhcos label

     apiVersion: v1
     kind: Pod
     metadata:
       name: <my-app>
     #...
     spec:
       nodeSelector:
         node.openshift.io/os_id: rhcos
     #...

     Copy to Clipboard

     The new pod will be created on RHCOS nodes, assuming the pod also has a controlling object.

Procedure

1. Identify nodes in your cluster that are running RHEL by entering the following command:

   $ oc get -l node.openshift.io/os_id=rhel

   Copy to Clipboard

   Example output

   NAME                        STATUS    ROLES     AGE       VERSION
   rhel-node1.example.com      Ready     worker    7h        v1.32.3
   rhel-node2.example.com      Ready     worker    7h        v1.32.3
   rhel-node3.example.com      Ready     worker    7h        v1.32.3

   Copy to Clipboard

2. Continue with the node removal process. RHEL nodes are not managed by the Machine API and have no compute machine sets associated with them. You must unschedule and drain the node before you manually delete it from the cluster.

   For more information on this process, see How to remove a worker node from Red Hat OpenShift Container Platform 4 UPI.