Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Staggered upgrade

As a storage administrator, you can upgrade Red Hat Ceph Storage components in phases rather than all at once. Starting with Red Hat Ceph Storage 5.2, the ceph orch upgrade command enables you to specify options to limit which daemons are upgraded by a single upgrade command.

Note

If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager (ceph-mgr) daemons. For more information on performing a staggered upgrade from previous releases, see Performing a staggered upgrade from previous releases.

Warning

The Ceph iSCSI gateway is removed from Red Hat Ceph Storage 6. Therefore, you need to manage the iSCSI LUNs before upgrading from Red Hat Ceph Storage 5 to Red Hat Ceph Storage 6.

When you upgrade a Red Hat Ceph Storage cluster from RHCS 5 to RHCS 6, RBD images that were exported through iSCSI are preserved, therefore data is not lost. However, because all iSCSI targets disappear with the upgrade, data is temporarily inaccessible. To recover data, you can map such RBD images with rbd device map command or export them to a file with rbd export command.

5.1. Staggered upgrade options
Copiar enlace

Starting with Red Hat Ceph Storage 5.2, the ceph orch upgrade command supports several options to upgrade cluster components in phases. The staggered upgrade options include:

  • --daemon_types: The --daemon_types option takes a comma-separated list of daemon types and will only upgrade daemons of those types. Valid daemon types for this option include mgr, mon, crash, osd, mds, rgw, rbd-mirror, cephfs-mirror, and nfs.
  • --services: The --services option is mutually exclusive with --daemon-types, only takes services of one type at a time, and will only upgrade daemons belonging to those services. For example, you cannot provide an OSD and RGW service simultaneously.
  • --hosts: You can combine the --hosts option with --daemon_types, --services, or use it on its own. The --hosts option parameter follows the same format as the command line options for orchestrator CLI placement specification.
  • --limit: The --limit option takes an integer greater than zero and provides a numerical limit on the number of daemons cephadm will upgrade. You can combine the --limit option with --daemon_types, --services, or --hosts. For example, if you specify to upgrade daemons of type osd on host01 with a limit set to 3, cephadm will upgrade up to three OSD daemons on host01.

5.1.1. Performing a staggered upgrade
Copiar enlace

As a storage administrator, you can use the ceph orch upgrade options to limit which daemons are upgraded by a single upgrade command.

Cephadm strictly enforces an order for the upgrade of daemons that is still present in staggered upgrade scenarios. The current upgrade order is:

  1. Ceph Manager nodes
  2. Ceph Monitor nodes
  3. Ceph-crash daemons
  4. Ceph OSD nodes
  5. Ceph Metadata Server (MDS) nodes
  6. Ceph Object Gateway (RGW) nodes
  7. Ceph RBD-mirror node
  8. CephFS-mirror node
  9. Ceph NFS nodes
Note

If you specify parameters that upgrade daemons out of order, the upgrade command blocks and notes which daemons you need to upgrade before you proceed.

Example

[ceph: root@host01 /]# ceph orch upgrade start --image  registry.redhat.io/rhceph/rhceph-6-rhel9:latest --hosts host02

Error EINVAL: Cannot start upgrade. Daemons with types earlier in upgrade order than daemons on given host need upgrading.
Please first upgrade mon.ceph-host01
Copy to Clipboard Toggle word wrap

Note

There is no required order for restarting the instances. Red Hat recommends restarting the instance pointing to the pool with primary images followed by the instance pointing to the mirrored pool.

Prerequisites

  • A cluster running Red Hat Ceph Storage 5.2 or later.
  • Root-level access to all the nodes.
  • At least two Ceph Manager nodes in the storage cluster: one active and one standby.

Procedure

  1. Log into the cephadm shell:

    Example

    [root@host01 ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. Ensure all the hosts are online and that the storage cluster is healthy:

    Example

    [ceph: root@host01 /]# ceph -s
    Copy to Clipboard Toggle word wrap

  3. Set the OSD noout, noscrub, and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster:

    Example

    [ceph: root@host01 /]# ceph osd set noout
[ceph: root@host01 /]# ceph osd set noscrub
[ceph: root@host01 /]# ceph osd set nodeep-scrub
    Copy to Clipboard Toggle word wrap

  4. Check service versions and the available target containers:

    Syntax

    ceph orch upgrade check IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch upgrade check registry.redhat.io/rhceph/rhceph-6-rhel9:latest
    Copy to Clipboard Toggle word wrap

    Note

    The image name is applicable for both Red Hat Enterprise Linux 8 and Red Hat Enterprise Linux 9.

  5. Upgrade the storage cluster:

    1. To upgrade specific daemon types on specific hosts:

      Syntax

      ceph orch upgrade start --image IMAGE_NAME --daemon-types DAEMON_TYPE1,DAEMON_TYPE2 --hosts HOST1,HOST2
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-6-rhel9:latest --daemon-types mgr,mon --hosts host02,host03
      Copy to Clipboard Toggle word wrap

    2. To specify specific services and limit the number of daemons to upgrade:

      Syntax

      ceph orch upgrade start --image IMAGE_NAME --services SERVICE1,SERVICE2 --limit LIMIT_NUMBER
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-6-rhel9:latest --services rgw.example1,rgw1.example2 --limit 2
      Copy to Clipboard Toggle word wrap

      Note

      In staggered upgrade scenarios, if using a limiting parameter, the monitoring stack daemons, including Prometheus and node-exporter, are refreshed after the upgrade of the Ceph Manager daemons. As a result of the limiting parameter, Ceph Manager upgrades take longer to complete. The versions of monitoring stack daemons might not change between Ceph releases, in which case, they are only redeployed.

      Note

      Upgrade commands with limiting parameters validates the options before beginning the upgrade, which can require pulling the new container image. As a result, the upgrade start command might take a while to return when you provide limiting parameters.

  6. To see which daemons you still need to upgrade, run the ceph orch upgrade check or ceph versions command:

    Example

    [ceph: root@host01 /]# ceph orch upgrade check --image registry.redhat.io/rhceph/rhceph-6-rhel9:latest
    Copy to Clipboard Toggle word wrap

  7. To complete the staggered upgrade, verify the upgrade of all remaining services:

    Syntax

    ceph orch upgrade start --image IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-6-rhel9:latest
    Copy to Clipboard Toggle word wrap

Verification

  • Verify the new IMAGE_ID and VERSION of the Ceph cluster:

    Example

    [ceph: root@host01 /]# ceph versions
[ceph: root@host01 /]# ceph orch ps
    Copy to Clipboard Toggle word wrap

    1. When the upgrade is complete, unset the noout, noscrub, and nodeep-scrub flags:

      Example

      [ceph: root@host01 /]# ceph osd unset noout
[ceph: root@host01 /]# ceph osd unset noscrub
[ceph: root@host01 /]# ceph osd unset nodeep-scrub
      Copy to Clipboard Toggle word wrap

5.1.2. Performing a staggered upgrade from previous releases
Copiar enlace

Starting with Red Hat Ceph Storage 5.2, you can perform a staggered upgrade on your storage cluster by providing the necessary arguments. If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager (ceph-mgr) daemons. Once you have upgraded the Ceph Manager daemons, you can pass the limiting parameters to complete the staggered upgrade.

Important

Verify you have at least two running Ceph Manager daemons before attempting this procedure.

Prerequisites

  • A cluster running Red Hat Ceph Storage 5.0 or later.
  • At least two Ceph Manager nodes in the storage cluster: one active and one standby.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. Determine which Ceph Manager is active and which are standby:

    Example

    [ceph: root@host01 /]# ceph -s
  cluster:
    id:     266ee7a8-2a05-11eb-b846-5254002d4916
    health: HEALTH_OK


  services:
    mon: 2 daemons, quorum host01,host02 (age 92s)
    mgr: host01.ndtpjh(active, since 16h), standbys: host02.pzgrhz
    Copy to Clipboard Toggle word wrap

  3. Manually upgrade each standby Ceph Manager daemon:

    Syntax

    ceph orch daemon redeploy mgr.ceph-HOST.MANAGER_ID --image IMAGE_ID
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch daemon redeploy mgr.ceph-host02.pzgrhz --image registry.redhat.io/rhceph/rhceph-6-rhel9:latest
    Copy to Clipboard Toggle word wrap

  4. Fail over to the upgraded standby Ceph Manager:

    Example

    [ceph: root@host01 /]# ceph mgr fail
    Copy to Clipboard Toggle word wrap

  5. Check that the standby Ceph Manager is now active:

    Example

    [ceph: root@host01 /]# ceph -s
  cluster:
    id:     266ee7a8-2a05-11eb-b846-5254002d4916
    health: HEALTH_OK


  services:
    mon: 2 daemons, quorum host01,host02 (age 1h)
    mgr: host02.pzgrhz(active, since 25s), standbys: host01.ndtpjh
    Copy to Clipboard Toggle word wrap

  6. Verify that the active Ceph Manager is upgraded to the new version:

    Syntax

    ceph tell mgr.ceph-HOST.MANAGER_ID version
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph tell mgr.host02.pzgrhz version
{
    "version": "16.2.8-12.el8cp",
    "release": "pacific",
    "release_type": "stable"
}
    Copy to Clipboard Toggle word wrap

  7. Repeat steps 2 - 6 to upgrade the remaining Ceph Managers to the new version.

  8. Check that all Ceph Managers are upgraded to the new version:

    Example

    [ceph: root@host01 /]# ceph mgr versions
{
    "ceph version 16.2.8-12.el8cp (600e227816517e2da53d85f2fab3cd40a7483372) pacific (stable)": 2
}
    Copy to Clipboard Toggle word wrap

  9. Once you upgrade all your Ceph Managers, you can specify the limiting parameters and complete the remainder of the staggered upgrade.
Volver arriba
Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2025 Red Hat