Updating clusters

1. The target version is stored in the spec.desiredUpdate.version field of the ClusterVersion resource, which may be managed through the web console or the CLI.
2. The Cluster Version Operator (CVO) detects that the desiredUpdate in the ClusterVersion resource differs from the current cluster version. Using graph data from the OpenShift Update Service, the CVO resolves the desired cluster version to a pull spec for the release image.
3. The CVO validates the integrity and authenticity of the release image. Red Hat publishes cryptographically-signed statements about published release images at predefined locations by using image SHA digests as unique and immutable release image identifiers. The CVO utilizes a list of built-in public keys to validate the presence and signatures of the statement matching the checked release image.
4. The CVO creates a job named version-$version-$hash in the openshift-cluster-version namespace. This job uses containers that are executing the release image, so the cluster downloads the image through the container runtime. The job then extracts the manifests and metadata from the release image to a shared volume that is accessible to the CVO.
5. The CVO validates the extracted manifests and metadata.
6. The CVO checks some preconditions to ensure that no problematic condition is detected in the cluster. Certain conditions can prevent updates from proceeding. These conditions are either determined by the CVO itself, or reported by individual cluster Operators that detect some details about the cluster that the Operator considers problematic for the update.
7. The CVO records the accepted release in status.desired and creates a status.history entry about the new update.
8. The CVO begins reconciling the manifests from the release image. Cluster Operators are updated in separate stages called Runlevels, and the CVO ensures that all Operators in a Runlevel finish updating before it proceeds to the next level.
9. Manifests for the CVO itself are applied early in the process. When the CVO deployment is applied, the current CVO pod stops, and a CVO pod that uses the new version starts. The new CVO proceeds to reconcile the remaining manifests.
10. The update proceeds until the entire control plane is updated to the new version. Individual cluster Operators might perform update tasks on their domain of the cluster, and while they do so, they report their state through the Progressing=True condition.
11. The Machine Config Operator (MCO) manifests are applied towards the end of the process. The updated MCO then begins updating the system configuration and operating system of every node. Each node might be drained, updated, and rebooted before it starts to accept workloads again.

1. Cordon and drain the node

   Note

   When a node is cordoned, workloads cannot be scheduled to it.

2. Update the system configuration and operating system (OS) of the node
3. Reboot the node
4. Uncordon the node

1. The MCO cordons nodes 1, 2, and 3, and begins to drain them.
2. Node 2 finishes draining, reboots, and becomes available again. The MCO cordons node 4 and begins draining it.
3. Node 1 finishes draining, reboots, and becomes available again. The MCO cordons node 5 and begins draining it.
4. Node 3 finishes draining, reboots, and becomes available again.
5. Node 5 finishes draining, reboots, and becomes available again.
6. Node 4 finishes draining, reboots, and becomes available again.

Procedure

1. Obtain the pull spec for the update that you want to apply by running the following command:

   $ oc adm upgrade

   The output of this command includes pull specs for the available updates similar to the following:

   Partial example output

   ...
   Recommended updates:
   
   VERSION IMAGE
   4.16.0  quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
   ...

2. Set a $RELEASE_IMAGE variable with the release image that you want to use by running the following command:

   $ RELEASE_IMAGE=<update_pull_spec>

   where <update_pull_spec> is the pull spec for the release image that you want to use. For example:

   quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032

3. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image by running the following command:

   $ oc adm release extract \
     --from=$RELEASE_IMAGE \
     --credentials-requests \
     --included \1
     --to=<path_to_directory_for_credentials_requests> 2

   1

   The --included parameter includes only the manifests that your specific cluster configuration requires for the target release.

   2

   Specify the path to the directory where you want to store the CredentialsRequest objects. If the specified directory does not exist, this command creates it.

   This command creates a YAML file for each CredentialsRequest object.

4. For each CredentialsRequest CR in the release image, ensure that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.

   Sample AWS CredentialsRequest object

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cloud-credential-operator-iam-ro
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - effect: Allow
         action:
         - iam:GetUser
         - iam:GetUserPolicy
         - iam:ListAccessKeys
         resource: "*"
     secretRef:
       name: cloud-credential-operator-iam-ro-creds
       namespace: openshift-cloud-credential-operator 1

   1

   This field indicates the namespace which must exist to hold the generated secret.

   The CredentialsRequest CRs for other platforms have a similar format with different platform-specific values.

5. For any CredentialsRequest CR for which the cluster does not already have a namespace with the name specified in spec.secretRef.namespace, create the namespace by running the following command:

   $ oc create namespace <component_namespace>

Procedure

1. Set a variable for the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(oc get clusterversion -o jsonpath={..desired.image})

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE \
     --file="/usr/bin/ccoctl.<rhel_version>" \1
     -a ~/.pull-secret

   1

   For <rhel_version>, specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid:

   • rhel8: Specify this value for hosts that use RHEL 8.
   • rhel9: Specify this value for hosts that use RHEL 9.

4. Change the permissions to make ccoctl executable by running the following command:

   $ chmod 775 ccoctl.<rhel_version>

Procedure

1. Use the ccoctl tool to process all CredentialsRequest objects by running the command for your cloud provider. The following commands process CredentialsRequest objects:

   Example 2.1. Amazon Web Services (AWS)

   $ ccoctl aws create-all \1
     --name=<name> \2
     --region=<aws_region> \3
     --credentials-requests-dir=<path_to_credentials_requests_directory> \4
     --output-dir=<path_to_ccoctl_output_dir> \5
     --create-private-s3-bucket 6

   1

   To create the AWS resources individually, use the "Creating AWS resources individually" procedure in the "Installing a cluster on AWS with customizations" content. This option might be useful if you need to review the JSON files that the ccoctl tool creates before modifying AWS resources, or if the process the ccoctl tool uses to create AWS resources automatically does not meet the requirements of your organization.

   2

   Specify the name used to tag any cloud resources that are created for tracking.

   3

   Specify the AWS region in which cloud resources will be created.

   4

   Specify the directory containing the files for the component CredentialsRequest objects.

   5

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   6

   Optional: By default, the ccoctl utility stores the OpenID Connect (OIDC) configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint. To store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL instead, use the --create-private-s3-bucket parameter.

   Example 2.2. Google Cloud Platform (GCP)

   $ ccoctl gcp create-all \
     --name=<name> \1
     --region=<gcp_region> \2
     --project=<gcp_project_id> \3
     --credentials-requests-dir=<path_to_credentials_requests_directory> \4
     --output-dir=<path_to_ccoctl_output_dir> 5

   1

   Specify the user-defined name for all created GCP resources used for tracking.

   2

   Specify the GCP region in which cloud resources will be created.

   3

   Specify the GCP project ID in which cloud resources will be created.

   4

   Specify the directory containing the files of CredentialsRequest manifests to create GCP service accounts.

   5

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   Example 2.3. IBM Cloud

   $ ccoctl ibmcloud create-service-id \
     --credentials-requests-dir=<path_to_credential_requests_directory> \1
     --name=<cluster_name> \2
     --output-dir=<installation_directory> \3
     --resource-group-name=<resource_group_name> 4

   1

   Specify the directory containing the files for the component CredentialsRequest objects.

   2

   Specify the name of the OpenShift Container Platform cluster.

   3

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   4

   Optional: Specify the name of the resource group used for scoping the access policies.

   Example 2.4. Microsoft Azure

   $ ccoctl azure create-managed-identities \
     --name <azure_infra_name> \1
     --output-dir ./output_dir \
     --region <azure_region> \2
     --subscription-id <azure_subscription_id> \3
     --credentials-requests-dir <path_to_directory_for_credentials_requests> \
     --issuer-url "${OIDC_ISSUER_URL}" \4
     --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \5
     --installation-resource-group-name "${AZURE_INSTALL_RG}" 6

   1

   The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value.

   2

   Specify the region of the existing cluster.

   3

   Specify the subscription ID of the existing cluster.

   4

   Specify the OIDC issuer URL from the existing cluster. You can obtain this value by running the following command:

   $ oc get authentication cluster \
     -o jsonpath \
     --template='{ .spec.serviceAccountIssuer }'

   5

   Specify the name of the resource group that contains the DNS zone.

   6

   Specify the Azure resource group name. You can obtain this value by running the following command:

   $ oc get infrastructure cluster \
     -o jsonpath \
     --template '{ .status.platformStatus.azure.resourceGroupName }'

   Example 2.5. Nutanix

   $ ccoctl nutanix create-shared-secrets \
     --credentials-requests-dir=<path_to_credentials_requests_directory> \1
     --output-dir=<ccoctl_output_dir> \2
     --credentials-source-filepath=<path_to_credentials_file> 3

   1

   Specify the path to the directory that contains the files for the component CredentialsRequests objects.

   2

   Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

   3

   Optional: Specify the directory that contains the credentials data YAML file. By default, ccoctl expects this file to be in <home_directory>/.nutanix/credentials.

   For each CredentialsRequest object, ccoctl creates the required provider resources and a permissions policy as defined in each CredentialsRequest object from the OpenShift Container Platform release image.

2. Apply the secrets to your cluster by running the following command:

   $ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

Procedure

1. Create YAML files with secrets for any CredentialsRequest custom resources that the new release image adds. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each CredentialsRequest object.

   Example 2.6. Sample AWS YAML files

   Sample AWS CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - effect: Allow
         action:
         - s3:CreateBucket
         - s3:DeleteBucket
         resource: "*"
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Sample AWS Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     aws_access_key_id: <base64_encoded_aws_access_key_id>
     aws_secret_access_key: <base64_encoded_aws_secret_access_key>

   Example 2.7. Sample Azure YAML files

   Note

   Global Azure and Azure Stack Hub use the same CredentialsRequest object and secret formats.

   Sample Azure CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Sample Azure Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     azure_subscription_id: <base64_encoded_azure_subscription_id>
     azure_client_id: <base64_encoded_azure_client_id>
     azure_client_secret: <base64_encoded_azure_client_secret>
     azure_tenant_id: <base64_encoded_azure_tenant_id>
     azure_resource_prefix: <base64_encoded_azure_resource_prefix>
     azure_resourcegroup: <base64_encoded_azure_resourcegroup>
     azure_region: <base64_encoded_azure_region>

   Example 2.8. Sample GCP YAML files

   Sample GCP CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: GCPProviderSpec
         predefinedRoles:
         - roles/iam.securityReviewer
         - roles/iam.roleViewer
         skipServiceCheck: true
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Sample GCP Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     service_account.json: <base64_encoded_gcp_service_account_file>

2. If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed permissions requirements, update the permissions as required.

Procedure

1. Log in to oc on the cluster as a user with the cluster-admin role.

2. Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field by running the following command:

   $ oc edit cloudcredential cluster

   Text to add

   ...
     metadata:
       annotations:
         cloudcredential.openshift.io/upgradeable-to: <version_number>
   ...

   Where <version_number> is the version that you are upgrading to, in the format x.y.z. For example, use 4.12.2 for OpenShift Container Platform 4.12.2.

   It may take several minutes after adding the annotation for the upgradeable status to change.

Verification

1. In the Administrator perspective of the web console, navigate to Administration → Cluster Settings.

2. To view the CCO status details, click cloud-credential in the Cluster Operators list.

   • If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.
3. When the Upgradeable status in the Conditions section is True, begin the OpenShift Container Platform upgrade.

1. Image validation stage
2. Build validation stage
3. Sign validation stage

Procedure

1. To list all the available MachineHealthCheck resources that you want to pause, run the following command:

   $ oc get machinehealthcheck -n openshift-machine-api

2. To pause the machine health checks, add the cluster.x-k8s.io/paused="" annotation to the MachineHealthCheck resource. Run the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused=""

   The annotated MachineHealthCheck resource resembles the following YAML file:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineHealthCheck
   metadata:
     name: example
     namespace: openshift-machine-api
     annotations:
       cluster.x-k8s.io/paused: ""
   spec:
     selector:
       matchLabels:
         role: worker
     unhealthyConditions:
     - type:    "Ready"
       status:  "Unknown"
       timeout: "300s"
     - type:    "Ready"
       status:  "False"
       timeout: "300s"
     maxUnhealthy: "40%"
   status:
     currentHealthy: 5
     expectedMachines: 5

   Important

   Resume the machine health checks after updating the cluster. To resume the check, remove the pause annotation from the MachineHealthCheck resource by running the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused-

Procedure

1. View the available updates and note the version number of the update that you want to apply:

   $ oc adm upgrade

   Example output

   Cluster version is 4.13.10
   Upstream is unset, so the cluster will use an appropriate default.
   Channel: stable-4.13 (available channels: candidate-4.13, candidate-4.14, fast-4.13, stable-4.13)
   Recommended updates:
     VERSION     IMAGE
     4.13.14     quay.io/openshift-release-dev/ocp-release@sha256:406fcc160c097f61080412afcfa7fd65284ac8741ac7ad5b480e304aba73674b
     4.13.13     quay.io/openshift-release-dev/ocp-release@sha256:d62495768e335c79a215ba56771ff5ae97e3cbb2bf49ed8fb3f6cefabcdc0f17
     4.13.12     quay.io/openshift-release-dev/ocp-release@sha256:73946971c03b43a0dc6f7b0946b26a177c2f3c9d37105441315b4e3359373a55
     4.13.11     quay.io/openshift-release-dev/ocp-release@sha256:e1c2377fdae1d063aaddc753b99acf25972b6997ab9a0b7e80cfef627b9ef3dd

   Note
   • If there are no recommended updates, updates that have known issues might still be available. See Updating along a conditional update path for more information.
   • For details and information on how to perform a Control Plane Only update, please refer to the Preparing to perform a Control Plane Only update page, listed in the Additional resources section.

2. Based on your organization requirements, set the appropriate update channel. For example, you can set your channel to stable-4.13 or fast-4.13. For more information about channels, refer to Understanding update channels and releases listed in the Additional resources section.

   $ oc adm upgrade channel <channel>

   For example, to set the channel to stable-4.16:

   $ oc adm upgrade channel stable-4.16

   Important

   For production clusters, you must subscribe to a stable-*, eus-*, or fast-* channel.

   Note

   When you are ready to move to the next minor version, choose the channel that corresponds to that minor version. The sooner the update channel is declared, the more effectively the cluster can recommend update paths to your target version. The cluster might take some time to evaluate all the possible updates that are available and offer the best update recommendations to choose from. Update recommendations can change over time, as they are based on what update options are available at the time.

   If you cannot see an update path to your target minor version, keep updating your cluster to the latest patch release for your current version until the next minor version is available in the path.

3. Apply an update:

   • To update to the latest version:

     $ oc adm upgrade --to-latest=true 1

   • To update to a specific version:

     $ oc adm upgrade --to=<version> 1

     1 1

     <version> is the update version that you obtained from the output of the oc adm upgrade command.

     Important

     When using oc adm upgrade --help, there is a listed option for --force. This is heavily discouraged, as using the --force option bypasses cluster-side guards, including release verification and precondition checks. Using --force does not guarantee a successful update. Bypassing guards put the cluster at risk.

4. Review the status of the Cluster Version Operator:

   $ oc adm upgrade

5. After the update completes, you can confirm that the cluster version has updated to the new version:

   $ oc adm upgrade

   Example output

   Cluster version is <version>
   
   Upstream is unset, so the cluster will use an appropriate default.
   Channel: stable-<version> (available channels: candidate-<version>, eus-<version>, fast-<version>, stable-<version>)
   
   No updates available. You may force an update to a specific release image, but doing so might not be supported and might result in downtime or data loss.

6. If you are updating your cluster to the next minor version, such as version X.y to X.(y+1), it is recommended to confirm that your nodes are updated before deploying workloads that rely on a new feature:

   $ oc get nodes

   Example output

   NAME                           STATUS   ROLES    AGE   VERSION
   ip-10-0-168-251.ec2.internal   Ready    master   82m   v1.29.4
   ip-10-0-170-223.ec2.internal   Ready    master   82m   v1.29.4
   ip-10-0-179-95.ec2.internal    Ready    worker   70m   v1.29.4
   ip-10-0-182-134.ec2.internal   Ready    worker   70m   v1.29.4
   ip-10-0-211-16.ec2.internal    Ready    master   82m   v1.29.4
   ip-10-0-250-100.ec2.internal   Ready    worker   69m   v1.29.4

Procedure

1. Set the OC_ENABLE_CMD_UPGRADE_STATUS environmental variable to true by running the following command:

   $ export OC_ENABLE_CMD_UPGRADE_STATUS=true

2. Run the oc adm upgrade status command:

   $ oc adm upgrade status

   Example 3.1. Example output for an update progressing successfully

   = Control Plane =
   Assessment:      Progressing
   Target Version:  4.14.1 (from 4.14.0)
   Completion:      97%
   Duration:        54m
   Operator Status: 32 Healthy, 1 Unavailable
   
   Control Plane Nodes
   NAME                                        ASSESSMENT    PHASE      VERSION   EST    MESSAGE
   ip-10-0-53-40.us-east-2.compute.internal    Progressing   Draining   4.14.0    +10m
   ip-10-0-30-217.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?
   ip-10-0-92-180.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?
   
   = Worker Upgrade =
   
   = Worker Pool =
   Worker Pool:     worker
   Assessment:      Progressing
   Completion:      0%
   Worker Status:   3 Total, 2 Available, 1 Progressing, 3 Outdated, 1 Draining, 0 Excluded, 0 Degraded
   
   Worker Pool Nodes
   NAME                                        ASSESSMENT    PHASE      VERSION   EST    MESSAGE
   ip-10-0-4-159.us-east-2.compute.internal    Progressing   Draining   4.14.0    +10m
   ip-10-0-20-162.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?
   ip-10-0-99-40.us-east-2.compute.internal    Outdated      Pending    4.14.0    ?
   
   = Worker Pool =
   Worker Pool:     infra
   Assessment:      Progressing
   Completion:      0%
   Worker Status:   1 Total, 0 Available, 1 Progressing, 1 Outdated, 1 Draining, 0 Excluded, 0 Degraded
   
   Worker Pool Node
   NAME                                             ASSESSMENT    PHASE      VERSION   EST    MESSAGE
   ip-10-0-4-159-infra.us-east-2.compute.internal   Progressing   Draining   4.14.0    +10m
   
   = Update Health =
   SINCE   LEVEL   IMPACT   MESSAGE
   14m4s   Info    None     Update is proceeding well

   With this information, you can make informed decisions on how to proceed with your update.

Procedure

1. To view the description of the update when it is not recommended because a risk might apply, run the following command:

   $ oc adm upgrade --include-not-recommended

2. If the cluster administrator evaluates the potential known risks and decides it is acceptable for the current cluster, then the administrator can waive the safety guards and proceed the update by running the following command:

   $ oc adm upgrade --allow-not-recommended --to <version> <.>

   <.> <version> is the update version that you obtained from the output of the previous command, which is supported but also has known issues or risks.

Procedure

1. Navigate to Administration → Cluster Settings, click version.

2. Click the YAML tab and then edit the upstream parameter value:

   Example output

     ...
     spec:
       clusterID: db93436d-7b05-42cc-b856-43e11ad2d31a
       upstream: '<update-server-url>' 1
     ...

   1

   The <update-server-url> variable specifies the URL for the update server.

   The default upstream is https://api.openshift.com/api/upgrades_info/v1/graph.

3. Click Save.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Compute → MachineHealthChecks.

3. To pause the machine health checks, add the cluster.x-k8s.io/paused="" annotation to each MachineHealthCheck resource. For example, to add the annotation to the machine-api-termination-handler resource, complete the following steps:

   1. Click the Options menu next to the machine-api-termination-handler and click Edit annotations.
   2. In the Edit annotations dialog, click Add more.
   3. In the Key and Value fields, add cluster.x-k8s.io/paused and "" values, respectively, and click Save.

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.16.

   Important

   For production clusters, you must subscribe to a stable-*, eus-* or fast-* channel.

   Note

   When you are ready to move to the next minor version, choose the channel that corresponds to that minor version. The sooner the update channel is declared, the more effectively the cluster can recommend update paths to your target version. The cluster might take some time to evaluate all the possible updates that are available and offer the best update recommendations to choose from. Update recommendations can change over time, as they are based on what update options are available at the time.

   If you cannot see an update path to your target minor version, keep updating your cluster to the latest patch release for your current version until the next minor version is available in the path.

   • If the Update status is not Updates available, you cannot update your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to, and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are updating your cluster to the next minor version, for example from version 4.10 to 4.11, confirm that your nodes are updated before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-*, eus-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

Procedure

1. From the web console, click Administration → Cluster settings page and review the contents of the Details tab.

2. You can enable the Include versions with known issues feature in the Select new version dropdown of the Update cluster modal to populate the dropdown list with conditional updates.

   Note

   If a version with known issues is selected, more information is provided with potential risks that are associated with the version.

3. Review the notification detailing the potential risks to updating.

1. Create custom machine config pools (MCP) based on the worker pool.

   Note

   You can change the maxUnavailable setting in an MCP to specify the percentage or the number of machines that can be updating at any given time. The default is 1.

   Warning

   The default setting for maxUnavailable is 1 for all the machine config pools in OpenShift Container Platform. It is recommended to not change this value and update one control plane node at a time. Do not change this value to 3 for the control plane pool.

2. Add a node selector to the custom MCPs. For each node that you do not want to update simultaneously with the rest of the cluster, add a matching label to the nodes. This label associates the node to the MCP.

   Important

   Do not remove the default worker label from the nodes. The nodes must have a role label to function properly in the cluster.

3. Pause the MCPs you do not want to update as part of the update process.
4. Perform the cluster update. The update process updates the MCPs that are not paused, including the control plane nodes.
5. Test your applications on the updated nodes to ensure they are working as expected.
6. Unpause one of the remaining MCPs, wait for the nodes in that pool to finish updating, and test the applications on those nodes. Repeat this process until all worker nodes are updated.
7. Optional: Remove the custom label from updated nodes and delete the custom MCPs.

Procedure

1. List the worker nodes in your cluster by running the following command:

   $ oc get -l 'node-role.kubernetes.io/master!=' -o 'jsonpath={range .items[*]}{.metadata.name}{"\n"}{end}' nodes

   Example output

   ci-ln-pwnll6b-f76d1-s8t9n-worker-a-s75z4
   ci-ln-pwnll6b-f76d1-s8t9n-worker-b-dglj2
   ci-ln-pwnll6b-f76d1-s8t9n-worker-c-lldbm

2. For each node that you want to delay, add a custom label to the node by running the following command:

   $ oc label node <node_name> node-role.kubernetes.io/<custom_label>=

   For example:

   $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary=

   Example output

   node/ci-ln-gtrwm8t-f76d1-spbl7-worker-a-xk76k labeled

3. Create the new MCP:

   1. Create an MCP YAML file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: workerpool-canary 1
      spec:
        machineConfigSelector:
          matchExpressions:
            - {
               key: machineconfiguration.openshift.io/role,
               operator: In,
               values: [worker,workerpool-canary] 2
              }
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/workerpool-canary: "" 3

      1

      Specify a name for the MCP.

      2

      Specify the worker and custom MCP name.

      3

      Specify the custom label you added to the nodes that you want in this pool.

   2. Create the MachineConfigPool object by running the following command:

      $ oc create -f <file_name>

      Example output

      machineconfigpool.machineconfiguration.openshift.io/workerpool-canary created

4. View the list of MCPs in the cluster and their current state by running the following command:

   $ oc get machineconfigpool

   Example output

   NAME              CONFIG                                                        UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master            rendered-master-b0bb90c4921860f2a5d8a2f8137c1867              True      False      False      3              3                   3                     0                      97m
   workerpool-canary rendered-workerpool-canary-87ba3dec1ad78cb6aecebf7fbb476a36   True      False      False      1              1                   1                     0                      2m42s
   worker            rendered-worker-87ba3dec1ad78cb6aecebf7fbb476a36              True      False      False      2              2                   2                     0                      97m

   The new machine config pool, workerpool-canary, is created and the number of nodes to which you added the custom label are shown in the machine counts. The worker MCP machine counts are reduced by the same number. It can take several minutes to update the machine counts. In this example, one node was moved from the worker MCP to the workerpool-canary MCP.

Procedure

1. Create a secondary MCP as described in the following two steps:

   1. Save the following configuration file as machineConfigPool.yaml.

      Example machineConfigPool YAML

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: worker-perf
      spec:
        machineConfigSelector:
          matchExpressions:
            - {
               key: machineconfiguration.openshift.io/role,
               operator: In,
               values: [worker,worker-perf]
              }
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker-perf: ""
      # ...

   2. Create the new machine config pool by running the following command:

      $ oc create -f machineConfigPool.yaml

      Example output

      machineconfigpool.machineconfiguration.openshift.io/worker-perf created

2. Add some machines to the secondary MCP. The following example labels the worker nodes worker-a, worker-b, and worker-c to the MCP worker-perf:

   $ oc label node worker-a node-role.kubernetes.io/worker-perf=''

   $ oc label node worker-b node-role.kubernetes.io/worker-perf=''

   $ oc label node worker-c node-role.kubernetes.io/worker-perf=''

3. Create a new MachineConfig for the MCP worker-perf as described in the following two steps:

   1. Save the following MachineConfig example as a file called new-machineconfig.yaml:

      Example MachineConfig YAML

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker-perf
        name: 06-kdump-enable-worker-perf
      spec:
        config:
          ignition:
            version: 3.2.0
          systemd:
            units:
            - enabled: true
              name: kdump.service
        kernelArguments:
          - crashkernel=512M
      # ...

   2. Apply the MachineConfig by running the following command:

      $ oc create -f new-machineconfig.yaml

4. Create the new canary MCP and add machines from the MCP you created in the previous steps. The following example creates an MCP called worker-perf-canary, and adds machines from the worker-perf MCP that you previosuly created.

   1. Label the canary worker node worker-a by running the following command:

      $ oc label node worker-a node-role.kubernetes.io/worker-perf-canary=''

   2. Remove the canary worker node worker-a from the original MCP by running the following command:

      $ oc label node worker-a node-role.kubernetes.io/worker-perf-

   3. Save the following file as machineConfigPool-Canary.yaml.

      Example machineConfigPool-Canary.yaml file

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: worker-perf-canary
      spec:
        machineConfigSelector:
          matchExpressions:
            - {
               key: machineconfiguration.openshift.io/role,
               operator: In,
               values: [worker,worker-perf,worker-perf-canary] 1
              }
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker-perf-canary: ""

      1

      Optional value. This example includes worker-perf-canary as an additional value. You can use a value in this way to configure members of an additional MachineConfig.

   4. Create the new worker-perf-canary by running the following command:

      $ oc create -f machineConfigPool-Canary.yaml

      Example output

      machineconfigpool.machineconfiguration.openshift.io/worker-perf-canary created

5. Check if the MachineConfig is inherited in worker-perf-canary.

   1. Verify that no MCP is degraded by running the following command:

      $ oc get mcp

      Example output

      NAME                  CONFIG                                                          UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master                rendered-master-2bf1379b39e22bae858ea1a3ff54b2ac                True      False      False      3              3                   3                     0                      5d16h
      worker                rendered-worker-b9576d51e030413cfab12eb5b9841f34                True      False      False      0              0                   0                     0                      5d16h
      worker-perf          rendered-worker-perf-b98a1f62485fa702c4329d17d9364f6a          True      False      False      2              2                   2                     0                      56m
      worker-perf-canary   rendered-worker-perf-canary-b98a1f62485fa702c4329d17d9364f6a   True      False      False      1              1                   1                     0                      44m

   2. Verify that the machines are inherited from worker-perf into worker-perf-canary.

      $ oc get nodes

      Example output

      NAME       STATUS   ROLES                        AGE     VERSION
      ...
      worker-a   Ready    worker,worker-perf-canary   5d15h   v1.27.13+e709aa5
      worker-b   Ready    worker,worker-perf          5d15h   v1.27.13+e709aa5
      worker-c   Ready    worker,worker-perf          5d15h   v1.27.13+e709aa5

   3. Verify that kdump service is enabled on worker-a by running the following command:

      $ systemctl status kdump.service

      Example output

      NAME       STATUS   ROLES                        AGE     VERSION
      ...
      kdump.service - Crash recovery kernel arming
           Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; preset: disabled)
           Active: active (exited) since Tue 2024-09-03 12:44:43 UTC; 10s ago
          Process: 4151139 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
         Main PID: 4151139 (code=exited, status=0/SUCCESS)

   4. Verify that the MCP has updated the crashkernel by running the following command:

      $ cat /proc/cmdline

      The output should include the updated crashekernel value, for example:

      Example output

      crashkernel=512M

6. Optional: If you are satisfied with the upgrade, you can return worker-a to worker-perf.

   1. Return worker-a to worker-perf by running the following command:

      $ oc label node worker-a node-role.kubernetes.io/worker-perf=''

   2. Remove worker-a from the canary MCP by running the following command:

      $ oc label node worker-a node-role.kubernetes.io/worker-perf-canary-

Procedure

1. Patch the MCP that you want paused by running the following command:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":true}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":true}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

Procedure

1. Patch the MCP that you want to unpause:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":false}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":false}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

2. Optional: Check the progress of the update by using one of the following options:

   1. Check the progress from the web console by clicking Administration → Cluster settings.

   2. Check the progress by running the following command:

      $ oc get machineconfigpools

3. Test your applications on the updated nodes to ensure that they are working as expected.
4. Repeat this process for any other paused MCPs, one at a time.

Procedure

1. For each node in a custom MCP, remove the custom label from the node by running the following command:

   $ oc label node <node_name> node-role.kubernetes.io/<custom_label>-

   For example:

   $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary-

   Example output

   node/ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz labeled

   The Machine Config Operator moves the nodes back to the original MCP and reconciles the node to the MCP configuration.

2. To ensure that node has been removed from the custom MCP, view the list of MCPs in the cluster and their current state by running the following command:

   $ oc get mcp

   Example output

   NAME                CONFIG                                                   UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master              rendered-master-1203f157d053fd987c7cbd91e3fbc0ed         True      False      False      3              3                   3                     0                      61m
   workerpool-canary   rendered-mcp-noupdate-5ad4791166c468f3a35cd16e734c9028   True      False      False      0              0                   0                     0                      21m
   worker              rendered-worker-5ad4791166c468f3a35cd16e734c9028         True      False      False      3              3                   3                     0                      61m

   When the node is removed from the custom MCP and moved back to the original MCP, it can take several minutes to update the machine counts. In this example, one node was moved from the removed workerpool-canary MCP to the worker MCP.

3. Optional: Delete the custom MCP by running the following command:

   $ oc delete mcp <mcp_name>

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.16.

   Important

   For production clusters, you must subscribe to a stable-*, eus-* or fast-* channel.

   Note

   When you are ready to move to the next minor version, choose the channel that corresponds to that minor version. The sooner the update channel is declared, the more effectively the cluster can recommend update paths to your target version. The cluster might take some time to evaluate all the possible updates that are available and offer the best update recommendations to choose from. Update recommendations can change over time, as they are based on what update options are available at the time.

   If you cannot see an update path to your target minor version, keep updating your cluster to the latest patch release for your current version until the next minor version is available in the path.

   • If the Update status is not Updates available, you cannot update your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to, and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are updating your cluster to the next minor version, for example from version 4.10 to 4.11, confirm that your nodes are updated before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-*, eus-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

   Important

   When you update a cluster that contains Red Hat Enterprise Linux (RHEL) worker machines, those workers temporarily become unavailable during the update process. You must run the update playbook against each RHEL machine as it enters the NotReady state for the cluster to finish updating.

Procedure

1. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   By default, the base OS RHEL with "Minimal" installation option enables firewalld service. Having the firewalld service enabled on your host prevents you from accessing OpenShift Container Platform logs on the worker. Do not enable firewalld later if you wish to continue accessing OpenShift Container Platform logs on the worker.

2. Enable the repositories that are required for OpenShift Container Platform 4.16:

   1. On the machine that you run the Ansible playbooks, update the required repositories:

      # subscription-manager repos --disable=rhocp-4.15-for-rhel-8-x86_64-rpms \
                                   --enable=rhocp-4.16-for-rhel-8-x86_64-rpms

      Important

      As of OpenShift Container Platform 4.11, the Ansible playbooks are provided only for RHEL 8. If a RHEL 7 system was used as a host for the OpenShift Container Platform 4.10 Ansible playbooks, you must either update the Ansible host to RHEL 8, or create a new Ansible host on a RHEL 8 system and copy over the inventories from the old Ansible host.

   2. On the machine that you run the Ansible playbooks, update the Ansible package:

      # yum swap ansible ansible-core

   3. On the machine that you run the Ansible playbooks, update the required packages, including openshift-ansible:

      # yum update openshift-ansible openshift-clients

   4. On each RHEL compute node, update the required repositories:

      # subscription-manager repos --disable=rhocp-4.15-for-rhel-8-x86_64-rpms \
                                   --enable=rhocp-4.16-for-rhel-8-x86_64-rpms

3. Update a RHEL worker machine:

   1. Review your Ansible inventory file at /<path>/inventory/hosts and update its contents so that the RHEL 8 machines are listed in the [workers] section, as shown in the following example:

      [all:vars]
      ansible_user=root
      #ansible_become=True
      
      openshift_kubeconfig_path="~/.kube/config"
      
      [workers]
      mycluster-rhel8-0.example.com
      mycluster-rhel8-1.example.com
      mycluster-rhel8-2.example.com
      mycluster-rhel8-3.example.com

   2. Change to the openshift-ansible directory:

      $ cd /usr/share/ansible/openshift-ansible

   3. Run the upgrade playbook:

      $ ansible-playbook -i /<path>/inventory/hosts playbooks/upgrade.yml 1

      1

      For <path>, specify the path to the Ansible inventory file that you created.

      Note

      The upgrade playbook only updates the OpenShift Container Platform packages. It does not update the operating system packages.

4. After you update all of the workers, confirm that all of your cluster nodes have updated to the new version:

   # oc get node

   Example output

   NAME                        STATUS                        ROLES    AGE    VERSION
   mycluster-control-plane-0   Ready                         master   145m   v1.29.4
   mycluster-control-plane-1   Ready                         master   145m   v1.29.4
   mycluster-control-plane-2   Ready                         master   145m   v1.29.4
   mycluster-rhel8-0           Ready                         worker   98m    v1.29.4
   mycluster-rhel8-1           Ready                         worker   98m    v1.29.4
   mycluster-rhel8-2           Ready                         worker   98m    v1.29.4
   mycluster-rhel8-3           Ready                         worker   98m    v1.29.4

5. Optional: Update the operating system packages that were not updated by the upgrade playbook. To update packages that are not on 4.16, use the following command:

   # yum update

   Note

   You do not need to exclude RPM packages if you are using the same RPM repository that you used when you installed 4.16.

1. Install the OpenShift CLI (oc) on all devices being used to retrieve and push release images.
2. Download the registry pull secret and add it to your cluster.

3. If you use the oc-mirror OpenShift CLI (oc) plugin:

   1. Install the oc-mirror plugin on all devices being used to retrieve and push release images.
   2. Create an image set configuration file for the plugin to use when determining which release images to mirror. You can edit this configuration file later to change which release images that the plugin mirrors.
   3. Mirror your targeted release images directly to a mirror registry, or to removable media and then to a mirror registry.
   4. Configure your cluster to use the resources generated by the oc-mirror plugin.
   5. Repeat these steps as needed to update your mirror registry.

4. If you use the oc adm release mirror command:

   1. Set environment variables that correspond to your environment and the release images you want to mirror.
   2. Mirror your targeted release images directly to a mirror registry, or to removable media and then to a mirror registry.
   3. Repeat these steps as needed to update your mirror registry.

1. Configure access to a secured registry.
2. Update the global cluster pull secret to access your mirror registry.
3. Install the OSUS Operator.
4. Create a graph data container image for the OpenShift Update Service.
5. Install the OSUS application and configure your clusters to use the OpenShift Update Service in your environment.
6. Perform a supported update procedure from the documentation as you would with a connected cluster.

Procedure

1. Verify that the RetrievedUpdates condition is True in the Cluster Version Operator (CVO) by running the following command:

   $ oc get clusterversion/version -o=jsonpath="{.status.conditions[?(.type=='RetrievedUpdates')].status}"

   If the RetrievedUpates condition is False, you can find supplemental information regarding the failure by using the following command:

   $ oc adm upgrade

   For more information about cluster version condition types, see Understanding cluster version condition types.

2. If the condition RetrievedUpdates is False, change the channel to stable-<4.y> or fast-<4.y> with the following command:

   $ oc adm upgrade channel <channel>

   After setting the channel, verify if RetrievedUpdates is True.

   For more information about channels, see Understanding update channels and releases.

3. Migrate to the multi-architecture payload with following command:

   $ oc adm upgrade --to-multi-arch

Procedure

1. To create a MachineConfig object inside of a config map in the management cluster, enter the following information:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <configmap-name>
     namespace: clusters
   data:
     config: |
       apiVersion: machineconfiguration.openshift.io/v1
       kind: MachineConfig
       metadata:
         labels:
           machineconfiguration.openshift.io/role: worker
         name: <machineconfig-name>
       spec:
         config:
           ignition:
             version: 3.2.0
           storage:
             files:
             - contents:
                 source: data:...
               mode: 420
               overwrite: true
               path: ${PATH} 1

   1

   Sets the path on the node where the MachineConfig object is stored.

2. After you add the object to the config map, you can apply the config map to the node pool as follows:

   $ oc edit nodepool <nodepool_name> --namespace <hosted_cluster_namespace>

   apiVersion: hypershift.openshift.io/v1alpha1
   kind: NodePool
   metadata:
   # ...
     name: nodepool-1
     namespace: clusters
   # ...
   spec:
     config:
     - name: ${configmap-name}
   # ...

1. Inspect the system status:

   # bootupctl status

   Example output for x86_64

   Component EFI
     Installed: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64
     Update: At latest version

   Example output for aarch64

   Component EFI
     Installed: grub2-efi-aa64-1:2.02-99.el8_4.1.aarch64,shim-aa64-15.4-2.el8_1.aarch64
     Update: At latest version

2. OpenShift Container Platform clusters initially installed on version 4.4 and older require an explicit adoption phase.

   If the system status is Adoptable, perform the adoption:

   # bootupctl adopt-and-update

   Example output

   Updated: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64

3. If an update is available, apply the update so that the changes take effect on the next reboot:

   # bootupctl update

   Example output

   Updated: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64

1. Create a Butane config file, 99-worker-bootupctl-update.bu, including the contents of the bootupctl-update.service systemd unit.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Example output

   variant: openshift
   version: 4.16.0
   metadata:
     name: 99-worker-chrony 1
     labels:
       machineconfiguration.openshift.io/role: worker 2
   systemd:
     units:
     - name: bootupctl-update.service
       enabled: true
       contents: |
         [Unit]
         Description=Bootupd automatic update
   
         [Service]
         ExecStart=/usr/bin/bootupctl update
         RemainAfterExit=yes
   
         [Install]
         WantedBy=multi-user.target

   1 2

   On control plane nodes, substitute master for worker in both of these locations.

2. Use Butane to generate a MachineConfig object file, 99-worker-bootupctl-update.yaml, containing the configuration to be delivered to the nodes:

   $ butane 99-worker-bootupctl-update.bu -o 99-worker-bootupctl-update.yaml

3. Apply the configurations in one of two ways:

   • If the cluster is not running yet, after you generate manifest files, add the MachineConfig object file to the <installation_directory>/openshift directory, and then continue to create the cluster.

   • If the cluster is already running, apply the file:

     $ oc apply -f ./99-worker-bootupctl-update.yaml