Updating clusters

Procedure

1. Upgrade any OLM Operators to versions that are compatible with both versions you are updating to.

2. Verify that all MachineConfigPools display a status of UPDATED and no MachineConfigPools display a status of UPDATING. View the status of all MachineConfigPools, run the following command:

   $ oc get mcp

   Example output

   Output is trimmed for clarity:

   NAME     CONFIG                                         	UPDATED   UPDATING
   master   rendered-master-ecbb9582781c1091e1c9f19d50cf836c       True  	  False
   worker   rendered-worker-00a3f0c68ae94e747193156b491553d5       True  	  False

3. Pause the MachineConfigPools you wish to skip reboots on, run the following commands:

   Note

   You cannot pause the master pool.

   $ oc patch mcp/worker --type merge --patch '{"spec":{"paused":true}}'

4. Change to the eus-4.10 channel, run the following command:

   $ oc adm upgrade channel eus-4.10

5. Update to 4.9, run the following command:

   $ oc adm upgrade --to-latest

   Example output

   Updating to latest version 4.9.18

6. Review the cluster version to ensure that the updates are complete by running the following command:

   $ oc get clusterversion

   Example output

   NAME  	  VERSION  AVAILABLE  PROGRESSING   SINCE   STATUS
   version   4.9.18   True       False         6m29s   Cluster version is 4.9.18

7. If necessary, upgrade OLM operators using the Administrator perspective on the web console.

8. Update to 4.10, run the following command:

   $ oc adm upgrade --to-latest

9. Review the cluster version to ensure that the updates are complete by running the following command:

   $ oc get clusterversion

   Example output

   NAME  	  VERSION  AVAILABLE  PROGRESSING   SINCE   STATUS
   version   4.10.1   True       False         6m29s   Cluster version is 4.10.1

10. Unpause all previously paused MachineConfigPools, run the following command:

    $ oc patch mcp/worker --type merge --patch '{"spec":{"paused":false}}'

    Note

    If pools are not unpaused, the cluster is not permitted to update to any future minors and maintenance tasks such as certificate rotation are inhibited. This puts the cluster at risk for future degradation.

11. Verify that your previously paused pools have updated and your cluster completed the update to 4.10, run the following command:

    $ oc get mcp

    Example output

    Output is trimmed for clarity:

    NAME 	   CONFIG                                            UPDATED     UPDATING
    master   rendered-master-52da4d2760807cb2b96a3402179a9a4c    True  	 False
    worker   rendered-worker-4756f60eccae96fb9dcb4c392c69d497    True 	 False

Procedure

1. Extract and examine the CredentialsRequest custom resource for the new release.

   The "Manually creating IAM" section of the installation content for your cloud provider explains how to obtain and use the credentials required for your cloud.

2. Update the manually maintained credentials on your cluster:

   • Create new secrets for any CredentialsRequest custom resources that are added by the new release image.
   • If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed their permissions requirements, update the permissions as required.

3. When all of the secrets are correct for the new release, indicate that the cluster is ready to upgrade:

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field:

      $ oc edit cloudcredential cluster

      Text to add

      ...
        metadata:
          annotations:
            cloudcredential.openshift.io/upgradeable-to: <version_number>
      ...

      Where <version_number> is the version you are upgrading to, in the format x.y.z. For example, 4.8.2 for OpenShift Container Platform 4.8.2.

      It may take several minutes after adding the annotation for the upgradeable status to change.

4. Verify that the CCO is upgradeable:

   1. In the Administrator perspective of the web console, navigate to Administration → Cluster Settings.
   2. To view the CCO status details, click cloud-credential in the Cluster Operators list.
   3. If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Compute → MachineHealthChecks.

3. To pause the machine health checks, add the cluster.x-k8s.io/paused="" annotation to each MachineHealthCheck resource. For example, to add the annotation to the machine-api-termination-handler resource, complete the following steps:

   1. Click the Options menu next to the machine-api-termination-handler and click Edit annotations.
   2. In the Edit annotations dialog, click Add more.
   3. In the Key and Value fields, add cluster.x-k8s.io/paused and "" values, respectively, and click Save.

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.9.

   Important

   For production clusters, you must subscribe to a stable-*, eus-* or fast-* channel.

   • If the Update status is not Updates available, you cannot upgrade your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are upgraded before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-*, eus-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

Procedure

1. Navigate to Administration → Cluster Settings, click version.

2. Click the YAML tab and then edit the upstream parameter value:

   Example output

     ...
     spec:
       clusterID: db93436d-7b05-42cc-b856-43e11ad2d31a
       upstream: '<update-server-url>' 1
     ...

   1

   The <update-server-url> variable specifies the URL for the update server.

   The default upstream is https://api.openshift.com/api/upgrades_info/v1/graph.

3. Click Save.

Procedure

1. To list all the available MachineHealthCheck resources that you want to pause, run the following command:

   $ oc get machinehealthcheck -n openshift-machine-api

2. To pause the machine health checks, add the cluster.x-k8s.io/paused="" annotation to the MachineHealthCheck resource. Run the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused=""

   The annotated MachineHealthCheck resource resembles the following YAML file:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineHealthCheck
   metadata:
     name: example
     namespace: openshift-machine-api
     annotations:
       cluster.x-k8s.io/paused: ""
   spec:
     selector:
       matchLabels:
         role: worker
     unhealthyConditions:
     - type:    "Ready"
       status:  "Unknown"
       timeout: "300s"
     - type:    "Ready"
       status:  "False"
       timeout: "300s"
     maxUnhealthy: "40%"
   status:
     currentHealthy: 5
     expectedMachines: 5

   Important

   Resume the machine health checks after updating the cluster. To resume the check, remove the pause annotation from the MachineHealthCheck resource by running the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused-

Procedure

1. Ensure that your cluster is available:

   $ oc get clusterversion

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.8.13     True        False         158m    Cluster version is 4.8.13

2. Review the current update channel information and confirm that your channel is set to stable-4.9:

   $ oc get clusterversion -o json|jq ".items[0].spec"

   Example output

   {
     "channel": "stable-4.9",
     "clusterID": "990f7ab8-109b-4c95-8480-2bd1deec55ff"
   }

   Important

   For production clusters, you must subscribe to a stable-*, eus-* or fast-* channel.

3. View the available updates and note the version number of the update that you want to apply:

   $ oc adm upgrade

   Example output

   Cluster version is 4.8.13
   
   Updates:
   
   VERSION IMAGE
   4.9.0   quay.io/openshift-release-dev/ocp-release@sha256:9c5f0df8b192a0d7b46cd5f6a4da2289c155fd5302dec7954f8f06c878160b8b

4. Apply an update:

   • To update to the latest version:

     $ oc adm upgrade --to-latest=true 1

   • To update to a specific version:

     $ oc adm upgrade --to=<version> 1

     1 1

     <version> is the update version that you obtained from the output of the previous command.

5. Review the status of the Cluster Version Operator:

   $ oc get clusterversion -o json|jq ".items[0].spec"

   Example output

   {
     "channel": "stable-4.9",
     "clusterID": "990f7ab8-109b-4c95-8480-2bd1deec55ff",
     "desiredUpdate": {
       "force": false,
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:9c5f0df8b192a0d7b46cd5f6a4da2289c155fd5302dec7954f8f06c878160b8b",
       "version": "4.9.0" 1
     }
   }

   1

   If the version number in the desiredUpdate stanza matches the value that you specified, the update is in progress.

6. Review the cluster version status history to monitor the status of the update. It might take some time for all the objects to finish updating.

   $ oc get clusterversion -o json|jq ".items[0].status.history"

   Example output

   [
     {
       "completionTime": null,
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:b8fa13e09d869089fc5957c32b02b7d3792a0b6f36693432acc0409615ab23b7",
       "startedTime": "2021-01-28T20:30:50Z",
       "state": "Partial",
       "verified": true,
       "version": "4.9.0"
     },
     {
       "completionTime": "2021-01-28T20:30:50Z",
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:b8fa13e09d869089fc5957c32b02b7d3792a0b6f36693432acc0409615ab23b7",
       "startedTime": "2021-01-28T17:38:10Z",
       "state": "Completed",
       "verified": false,
       "version": "4.8.13"
     }
   ]

   The history contains a list of the most recent versions applied to the cluster. This value is updated when the CVO applies an update. The list is ordered by date, where the newest update is first in the list. Updates in the history have state Completed if the rollout completed and Partial if the update failed or did not complete.

7. After the update completes, you can confirm that the cluster version has updated to the new version:

   $ oc get clusterversion

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   SINCE     STATUS
   version   4.9.0       True        False         2m        Cluster version is 4.9.0

   Note

   If the oc get clusterversion command displays the following error while the PROGRESSING status is True, you can ignore the error.

   NAME    VERSION AVAILABLE PROGRESSING SINCE STATUS
   version 4.10.26 True      True        24m   Unable to apply 4.11.0-rc.7: an unknown error has occurred: MultipleErrors

8. If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are updated before deploying workloads that rely on a new feature:

   $ oc get nodes

   Example output

   NAME                           STATUS   ROLES    AGE   VERSION
   ip-10-0-168-251.ec2.internal   Ready    master   82m   v1.22.1
   ip-10-0-170-223.ec2.internal   Ready    master   82m   v1.22.1
   ip-10-0-179-95.ec2.internal    Ready    worker   70m   v1.22.1
   ip-10-0-182-134.ec2.internal   Ready    worker   70m   v1.22.1
   ip-10-0-211-16.ec2.internal    Ready    master   82m   v1.22.1
   ip-10-0-250-100.ec2.internal   Ready    worker   69m   v1.22.1

1. Create MCPs based on the worker pool. The number of nodes in each MCP depends on a few factors, such as your maintenance window duration for each MCP, and the amount of reserve capacity, meaning extra worker nodes, available in your cluster.

   Note

   You can change the maxUnavailable setting in an MCP to specify the percentage or the number of machines that can be updating at any given time. The default is 1.

2. Add a node selector to the custom MCPs. For each node that you do not want to update simultaneously with the rest of the cluster, add a matching label to the nodes. This label associates the node to the MCP.

   Note

   Do not remove the default worker label from the nodes. The nodes must have a role label to function properly in the cluster.

3. Pause the MCPs you do not want to update as part of the update process.

   Note

   Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the Understand CA cert auto renewal in Red Hat OpenShift 4 to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple oc commands, including but not limited to oc debug, oc logs, oc exec, and oc attach.

4. Perform the cluster update. The update process updates the MCPs that are not paused, including the control plane nodes.
5. Test the applications on the updated nodes to ensure they are working as expected.
6. Unpause the remaining MCPs one-by-one and test the applications on those nodes until all worker nodes are updated. Unpausing an MCP starts the update process for the nodes associated with that MCP. You can check the progress of the update from the web console by clicking Administration → Cluster settings. Or, use the oc get machineconfigpools CLI command.
7. Optionally, remove the custom label from updated nodes and delete the custom MCPs.

1. Create an MCP from a worker node.

   1. List the worker nodes in your cluster.

      $ oc get -l 'node-role.kubernetes.io/master!=' -o 'jsonpath={range .items[*]}{.metadata.name}{"\n"}{end}' nodes

      Example output

      ci-ln-pwnll6b-f76d1-s8t9n-worker-a-s75z4
      ci-ln-pwnll6b-f76d1-s8t9n-worker-b-dglj2
      ci-ln-pwnll6b-f76d1-s8t9n-worker-c-lldbm

   2. For the nodes you want to delay, add a custom label to the node:

      $ oc label node <node name> node-role.kubernetes.io/<custom-label>=

      For example:

      $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary=

      Example output

      node/ci-ln-gtrwm8t-f76d1-spbl7-worker-a-xk76k labeled

   3. Create the new MCP:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: workerpool-canary 1
      spec:
        machineConfigSelector:
          matchExpressions: 2
            - {
               key: machineconfiguration.openshift.io/role,
               operator: In,
               values: [worker,workerpool-canary]
              }
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/workerpool-canary: "" 3

      1

      Specify a name for the MCP.

      2

      Specify the worker and custom MCP name.

      3

      Specify the custom label you added to the nodes that you want in this pool.

      $ oc create -f <file_name>

      Example output

      machineconfigpool.machineconfiguration.openshift.io/workerpool-canary created

   4. View the list of MCPs in the cluster and their current state:

      $ oc get machineconfigpool

      Example output

      NAME              CONFIG                                                        UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master            rendered-master-b0bb90c4921860f2a5d8a2f8137c1867              True      False      False      3              3                   3                     0                      97m
      workerpool-canary rendered-workerpool-canary-87ba3dec1ad78cb6aecebf7fbb476a36   True      False      False      1              1                   1                     0                      2m42s
      worker            rendered-worker-87ba3dec1ad78cb6aecebf7fbb476a36              True      False      False      2              2                   2                     0                      97m

      The new machine config pool, workerpool-canary, is created and the number of nodes to which you added the custom label are shown in the machine counts. The worker MCP machine counts are reduced by the same number. It can take several minutes to update the machine counts. In this example, one node was moved from the worker MCP to the workerpool-canary MCP.

1. Patch the MCP that you want paused:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":true}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":true}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

1. Patch the MCP that you want to unpause:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":false}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":false}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

   You can check the progress of the update by using the oc get machineconfigpools command.

2. Test your applications on the updated nodes to ensure that they are working as expected.
3. Unpause any other paused MCPs one-by-one and verify that your applications work.

1. Remove the custom label from the node.

   $ oc label node <node_name> node-role.kubernetes.io/<custom-label>-

   For example:

   $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary-

   Example output

   node/ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz labeled

   The MCO moves the nodes back to the original MCP and reconciles the node to the MCP configuration.

2. View the list of MCPs in the cluster and their current state:

   $oc get mcp

   NAME                CONFIG                                                   UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master              rendered-master-1203f157d053fd987c7cbd91e3fbc0ed         True      False      False      3              3                   3                     0                      61m
   workerpool-canary   rendered-mcp-noupdate-5ad4791166c468f3a35cd16e734c9028   True      False      False      0              0                   0                     0                      21m
   worker              rendered-worker-5ad4791166c468f3a35cd16e734c9028         True      False      False      3              3                   3                     0                      61m

   The node is removed from the custom MCP and moved back to the original MCP. It can take several minutes to update the machine counts. In this example, one node was moved from the removed workerpool-canary MCP to the `worker`MCP.

3. Optional: Delete the custom MCP:

   $ oc delete mcp <mcp_name>

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.9.

   Important

   For production clusters, you must subscribe to a stable-*, eus-* or fast-* channel.

   • If the Update status is not Updates available, you cannot upgrade your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are upgraded before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-*, eus-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

   Note

   When you update a cluster that contains Red Hat Enterprise Linux (RHEL) worker machines, those workers temporarily become unavailable during the update process. You must run the upgrade playbook against each RHEL machine as it enters the NotReady state for the cluster to finish updating.

Procedure

1. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   By default, the base OS RHEL with "Minimal" installation option enables firewalld serivce. Having the firewalld service enabled on your host prevents you from accessing OpenShift Container Platform logs on the worker. Do not enable firewalld later if you wish to continue accessing OpenShift Container Platform logs on the worker.

2. Enable the repositories that are required for OpenShift Container Platform 4.9:

   1. On the machine that you run the Ansible playbooks, update the required repositories:

      # subscription-manager repos --disable=rhel-7-server-ose-4.8-rpms \
                                   --enable=rhel-7-server-ansible-2.9-rpms \
                                   --enable=rhel-7-server-ose-4.9-rpms

   2. On the machine that you run the Ansible playbooks, update the required packages, including openshift-ansible:

      # yum update openshift-ansible openshift-clients

   3. On each RHEL compute node, update the required repositories:

      # subscription-manager repos --disable=rhel-7-server-ose-4.8-rpms \
                                   --enable=rhel-7-server-ose-4.9-rpms  \
                                   --enable=rhel-7-fast-datapath-rpms   \
                                   --enable=rhel-7-server-optional-rpms

3. Update a RHEL worker machine:

   1. Review the current node status to determine which RHEL worker to update:

      # oc get node

      Example output

      NAME                        STATUS                        ROLES    AGE    VERSION
      mycluster-control-plane-0   Ready                         master   145m   v1.22.1
      mycluster-control-plane-1   Ready                         master   145m   v1.22.1
      mycluster-control-plane-2   Ready                         master   145m   v1.22.1
      mycluster-rhel7-0           NotReady,SchedulingDisabled   worker   98m    v1.22.1
      mycluster-rhel7-1           Ready                         worker   98m    v1.22.1
      mycluster-rhel7-2           Ready                         worker   98m    v1.22.1
      mycluster-rhel7-3           Ready                         worker   98m    v1.22.1

      Note which machine has the NotReady,SchedulingDisabled status.

   2. Review your Ansible inventory file at /<path>/inventory/hosts and update its contents so that only the machine with the NotReady,SchedulingDisabled status is listed in the [workers] section, as shown in the following example:

      [all:vars]
      ansible_user=root
      #ansible_become=True
      
      openshift_kubeconfig_path="~/.kube/config"
      
      [workers]
      mycluster-rhel7-0.example.com

   3. Change to the openshift-ansible directory:

      $ cd /usr/share/ansible/openshift-ansible

   4. Run the upgrade playbook:

      $ ansible-playbook -i /<path>/inventory/hosts playbooks/upgrade.yml 1

      1

      For <path>, specify the path to the Ansible inventory file that you created.

      Note

      The upgrade playbook only upgrades the OpenShift Container Platform packages. It does not update the operating system packages.

4. Follow the process in the previous step to update each RHEL worker machine in your cluster.

5. After you update all of the workers, confirm that all of your cluster nodes have updated to the new version:

   # oc get node

   Example output

   NAME                        STATUS                        ROLES    AGE    VERSION
   mycluster-control-plane-0   Ready                         master   145m   v1.22.1
   mycluster-control-plane-1   Ready                         master   145m   v1.22.1
   mycluster-control-plane-2   Ready                         master   145m   v1.22.1
   mycluster-rhel7-0           NotReady,SchedulingDisabled   worker   98m    v1.22.1
   mycluster-rhel7-1           Ready                         worker   98m    v1.22.1
   mycluster-rhel7-2           Ready                         worker   98m    v1.22.1
   mycluster-rhel7-3           Ready                         worker   98m    v1.22.1

6. Optional: Update the operating system packages that were not updated by the upgrade playbook. To update packages that are not on 4.9, use the following command:

   # yum update

   Note

   You do not need to exclude RPM packages if you are using the same RPM repository that you used when you installed 4.9.