Chapter 4. Staggered upgrade

1. Ceph Manager nodes
2. Ceph Monitor nodes
3. Ceph-crash daemons
4. Ceph OSD nodes
5. Ceph Metadata Server (MDS) nodes
6. Ceph Object Gateway (RGW) nodes
7. Ceph RBD-mirror node
8. CephFS-mirror node
9. Ceph NFS nodes

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Ensure all the hosts are online and that the storage cluster is healthy:

   Example

   [ceph: root@host01 /]# ceph -s

3. Set the OSD noout, noscrub, and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster:

   Example

   [ceph: root@host01 /]# ceph osd set noout
   [ceph: root@host01 /]# ceph osd set noscrub
   [ceph: root@host01 /]# ceph osd set nodeep-scrub

4. Check service versions and the available target containers:

   Syntax

   ceph orch upgrade check IMAGE_NAME

   Example

   [ceph: root@host01 /]# ceph orch upgrade check registry.redhat.io/rhceph/rhceph-8-rhel9:latest

5. Upgrade the storage cluster in one of the following ways:

   • Upgrade specific daemon types on specific hosts.

     Syntax

     ceph orch upgrade start --image IMAGE_NAME --daemon-types DAEMON_TYPE1,DAEMON_TYPE2 --hosts HOST1,HOST2

     Example

     [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-8-rhel9:latest --daemon-types mgr,mon --hosts host02,host03

   • Specify specific services and limit the number of daemons to upgrade.

     Note

     In staggered upgrade scenarios, if using a limiting parameter, the monitoring stack daemons, including Prometheus and node-exporter, are refreshed after the upgrade of the Ceph Manager daemons. As a result of the limiting parameter, Ceph Manager upgrades take longer to complete. The versions of monitoring stack daemons might not change between Ceph releases, in which case, they are only redeployed.

     Note

     Upgrade commands with limiting parameters validates the options before beginning the upgrade, which can require pulling the new container image. As a result, the upgrade start command might take a while to return when you provide limiting parameters.

     Syntax

     ceph orch upgrade start --image IMAGE_NAME --services SERVICE1,SERVICE2 --limit LIMIT_NUMBER

     Example

     [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-8-rhel9:latest --services rgw.example1,rgw1.example2 --limit 2

   • Valid on {stroage-product} 8.1 or later only. Specify specific topological labels to run the upgrade on specific hosts.

     Note

     Using topological labels does not allowe changing from the required daemon upgrade order.

     Syntax

     ceph orch upgrade start --topological-labels _TOPOLOGICAL_LABEL_

     Example

     [ceph: root@host01 /]# ceph orch upgrade start --topological-labels datacenter=A

   • Valid on Red Hat Ceph Storage 8.1 or later only with topological labels. Specify specific topological labels to run the upgrade on specific hosts, with the --daemon-types option.

     Use the --daemon-types option for hosts with a similar scenario to the following example:

     A host exists with both Monitor and OSD daemons that match datacenter=A but nonupgraded Monitor daemons on host that do not match datacenter=A. In this case, you only want to upgrade the Monitor daemons on the datacenter=A host, to not break the upgrade order. In this scenario, use the following command to upgrade only the Monitor daemons on datacenter=A, but not the OSD daemons.

     Syntax

     ceph orch upgrade start _IMAGE_NAME_ --daemon-types _DAEMON_TYPE1_ --topological-labels _TOPOLOGICAL_LABEL_

     Example

     [ceph: root@host01 /]# ceph orch upgrade start registry.redhat.io/rhceph/rhceph-8-rhel9:latest --daemon-types mon --topological-labels datacenter=A

     Use the ceph orch upgrade status command to verify what was selected for upgrade. View the "which" field in the output.

     Example

     [ceph: root@host01 /]# ceph orch upgrade status
     {
         "in_progress": true,
         "target_image": "cp.icr.io/cp/ibm-ceph/ceph-8-rhel9:latest",
         "services_complete": [
             "mon"
         ],
         "which": "Upgrading daemons of type(s) mon on host(s) host01",
         "progress": "1/1 daemons upgraded",
         "message": "Currently upgrading mon daemons",
         "is_paused": false
     }

6. To see which daemons you still need to upgrade, run the ceph orch upgrade check or ceph versions command:

   Example

   [ceph: root@host01 /]# ceph orch upgrade check --image registry.redhat.io/rhceph/rhceph-8-rhel9:latest

7. To complete the staggered upgrade, verify the upgrade of all remaining services:

   Syntax

   ceph orch upgrade start --image IMAGE_NAME

   Example

   [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-8-rhel9:latest

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Determine which Ceph Manager is active and which are standby:

   Example

   [ceph: root@host01 /]# ceph -s
     cluster:
       id:     266ee7a8-2a05-11eb-b846-5254002d4916
       health: HEALTH_OK
   
   
     services:
       mon: 2 daemons, quorum host01,host02 (age 92s)
       mgr: host01.ndtpjh(active, since 16h), standbys: host02.pzgrhz

3. Manually upgrade each standby Ceph Manager daemon:

   Syntax

   ceph orch daemon redeploy mgr.ceph-HOST.MANAGER_ID --image IMAGE_ID

   Example

   [ceph: root@host01 /]# ceph orch daemon redeploy mgr.ceph-host02.pzgrhz --image registry.redhat.io/rhceph/rhceph-8-rhel9:latest

4. Fail over to the upgraded standby Ceph Manager:

   Example

   [ceph: root@host01 /]# ceph mgr fail

5. Check that the standby Ceph Manager is now active:

   Example

   [ceph: root@host01 /]# ceph -s
     cluster:
       id:     266ee7a8-2a05-11eb-b846-5254002d4916
       health: HEALTH_OK
   
   
     services:
       mon: 2 daemons, quorum host01,host02 (age 1h)
       mgr: host02.pzgrhz(active, since 25s), standbys: host01.ndtpjh

6. Verify that the active Ceph Manager is upgraded to the new version:

   Syntax

   ceph tell mgr.ceph-HOST.MANAGER_ID version

   Example

   [ceph: root@host01 /]# ceph tell mgr.host02.pzgrhz version
   {
       "version": "18.2.0-128.el8cp",
       "release": "reef",
       "release_type": "stable"
   }

7. Repeat steps 2 - 6 to upgrade the remaining Ceph Managers to the new version.

8. Check that all Ceph Managers are upgraded to the new version:

   Example

   [ceph: root@host01 /]# ceph mgr versions
   {
       "ceph version 18.2.0-128.el8cp (600e227816517e2da53d85f2fab3cd40a7483372) pacific (stable)": 2
   }

9. Once you upgrade all your Ceph Managers, you can specify the limiting parameters and complete the remainder of the staggered upgrade.