Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 16. Upgrading Red Hat Ceph Storage 5 to 6

You can upgrade the Red Hat Ceph Storage cluster from Release 5 to 6 after all other upgrade tasks are completed.

Prerequisites

  • The upgrade from Red Hat OpenStack Platform 16.2 to 17.1 is complete.
  • All Controller nodes and Red Hat Ceph Storage nodes are upgraded to Red Hat Enterprise Linux (RHEL) 9.2. For more information, see Upgrading the control plane operating system.
  • In HCI environments, all Compute nodes must also be upgraded to RHEL 9.2. For more information, see Upgrading the Compute node operating system.
  • The current Red Hat Ceph Storage 5 cluster is healthy.

Perform the following tasks if Red Hat Ceph Storage is director-deployed in your environment.

16.1.1. Updating the cephadm client
Copy link

Before you upgrade the Red Hat Ceph Storage cluster, you must update the cephadm package in the overcloud nodes to Release 6.

Prerequisites

  • Log in to a Controller node and confirm that the health status of the Red Hat Ceph Storage cluster is HEALTH_OK: 

    $ sudo cephadm shell -- ceph -s

    If the status is not HEALTH_OK, correct any issues before continuing with this procedure. For more information about troubleshooting Red Hat Ceph Storage 5, see Troubleshooting Guide.

Procedure

  1. Create a playbook to enable the Red Hat Ceph Storage (tool only) repositories in the Controller nodes. It should contain the following information: 

    - hosts: all
  gather_facts: false
  tasks:
    - name: Enable RHCS 6 tools repo
      ansible.builtin.command: |
          subscription-manager repos --disable=rhceph-5-tools-for-rhel-9-x86_64-rpms --enable=rhceph-6-tools-for-rhel-9-x86_64-rpms
      become: true
    - name: Update cephadm
      ansible.builtin.package:
        name: cephadm
        state: latest
      become: true

  2. Run the playbook:

    ansible-playbook -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml <playbook_file_name> --limit <controller_role>

    • Replace <stack> with the name of your stack.
    • Replace <playbook_file_name> with the name of the playbook created in the previous step.
    • Replace <controller_role> with the role applied to Controller nodes.
    • Use the --limit option to apply the content to Controller nodes only.
  3. Log in to a Controller node.

  4. Verify that the cephadm package is updated to Release 6:

    $ sudo dnf info cephadm | grep -i version

The container-image-prepare.yaml file contains the ContainerImagePrepare parameter and defines the Red Hat Ceph Storage containers. This file is used by the tripleo-container-image prepare command to define the rules for obtaining container images for the undercloud and overcloud. Update this file with the correct image version before updating your environment.

Procedure

  1. Locate your container preparation file. The default name of this file is containers-prepare-parameter.yaml.
  2. Edit the container preparation file.

  3. Locate the ceph_tag parameter. The current entry should be similar to the following example: 

    ceph_namespace: registry.redhat.io
ceph_image: rhceph-6-rhel9
ceph_tag: '5'

  4. Update the ceph_tag parameter for Red Hat Ceph Storage 6: 

    ceph_namespace: registry.redhat.io
ceph_image: rhceph-6-rhel9
ceph_tag: '6'

  5. Edit the containers-image-prepare.yaml file and replace the Red Hat Ceph monitoring stack container related parameters with the following content: 

    ceph_alertmanager_image: ose-prometheus-alertmanager
ceph_alertmanager_namespace: registry.redhat.io/openshift4
ceph_alertmanager_tag: v4.12
ceph_grafana_image: rhceph-6-dashboard-rhel9
ceph_grafana_namespace: registry.redhat.io/rhceph
ceph_grafana_tag: latest
ceph_node_exporter_image: ose-prometheus-node-exporter
ceph_node_exporter_namespace: registry.redhat.io/openshift4
ceph_node_exporter_tag: v4.12
ceph_prometheus_image: ose-prometheus
ceph_prometheus_namespace: registry.redhat.io/openshift4
ceph_prometheus_tag: v4.12
  6. Save the file.

16.1.3. Running the container image prepare
Copy link

Complete the container image preparation process by running the director container preparation command. This prepares all container image configurations for the overcloud and retrieves the latest Red Hat Ceph Storage 6 container image.

Note

If you are using Red Hat Satellite Server to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment, do not perform this procedure. Update Satellite to include the Red Hat Ceph Storage 6 container image and update your containers-prepare-parameter.yaml file to reference the URL of the container image that is hosted on the Satellite server.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the stackrc undercloud credentials file: 

    $ source ~/stackrc

  3. Run the container preparation command:

    $ openstack tripleo container image prepare -e <container_preparation_file>

    • Replace <container_preparation_file> with the name of your file. The default file is containers-prepare-parameter.yaml.

  4. Verify that the new Red Hat Ceph Storage image is present in the undercloud registry:

    $ openstack tripleo container image list -f value | awk -F '//' '/ceph/ {print $2}'

  5. Verify that the new Red Hat Ceph Storage monitoring stack images are present in the undercloud registry:

    $ openstack tripleo container image list -f value | awk -F '//' '/dashboard|grafana|prometheus|alertmanager|node-exporter/ {print $2}'

Procedure

  1. Log in to a Controller node.

  2. List the current images from the Ceph Manager configuration: 

    $ sudo cephadm shell -- ceph config dump | grep image

  3. Update the Ceph Manager configuration for the monitoring stack services to use Red Hat Ceph Storage 6 images: 

    $ sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_alertmanager <alertmanager_image>
$ sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_grafana <grafana_image>
$ sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_node_exporter <node_exporter_image>
$ sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_prometheus <prometheus_image>
    • Replace <alertmanager_image> with the new alertmanager image.
    • Replace <grafana_image> with the new grafana image.
    • Replace <node_exporter_image> with the new node exporter image.

    • Replace <prometheus_image> with the new prometheus image.

      The following is an example of the alert manager update command: 

      $ sudo cephadm shell -- ceph config set mgr mgr/cephadm/container_image_alertmanager undercloud-0.ctlplane.redhat.local:8787/rh-osbs/openshift-ose-prometheus-alertmanager:v4.12

  4. Verify that the new image references are updated in the Red Hat Ceph Storage cluster: 

    $ sudo cephadm shell -- ceph config dump | grep image

Upgrade to Red Hat Ceph Storage 6 by using the Orchestrator capabilities of the cephadm command.

Prerequisities

  • On a Monitor or Controller node that is running the ceph-mon service, confirm the Red Hat Ceph Storage cluster status: 

    $ sudo cephadm shell -- ceph status

    This command returns one of three responses:

    • HEALTH_OK - The cluster is healthy. Proceed with the cluster upgrade.
    • HEALTH_WARN - The cluster is unhealthy. Do not proceed with the cluster upgrade before the blocking issues are resolved. For troubleshooting guidance, see Red Hat Ceph Storage 5 Troubleshooting Guide.
    • HEALTH_ERR - The cluster is unhealthy. Do not proceed with the cluster upgrade before the blocking issues are resolved. For troubleshooting guidance, see Red Hat Ceph Storage 5 Troubleshooting Guide.

Procedure

  1. Log in to a Controller node.

  2. Upgrade the cluster to the latest Red Hat Ceph Storage version by using Upgrade a Red Hat Ceph Storage cluster using cephadm in the Red Hat Ceph Storage 6 Upgrade Guide.

    Important

    Steps 1 to 4 of the upgrade procedure presented in Upgrading the Red Hat Ceph Storage cluster are not necessary when upgrading a Red Hat Ceph Storage cluster deployed with Red Hat OpenStack by director. Start at step 5 of the procedure.

  3. Wait until the Red Hat Ceph Storage container upgrade completes. Monitor the status of the upgrade by using the following command: 

    $ sudo cephadm shell -- ceph orch upgrade status

When Red Hat Ceph Storage is upgraded from Release 4 to 5, NFS Ganesha is not adopted by the Orchestrator. This means it remains under director control and must be moved manually to Release 6.

Red Hat Ceph Storage 5 based NFS Ganesha with a Red Hat Ceph Storage 6 cluster is only supported during the upgrade period. Once you upgrade the Red Hat Ceph Storage cluster to 6, you must upgrade NFS Ganesha to use a Release 6 based container image.

Note

This procedure only applies to environments that are using the Shared File Systems service (manila) with CephFS NFS. Upgradng the Red Hat Ceph Storage container for NFS Ganesha is mandatory in these environments.

Procedure

  1. Log in to a Controller node.

  2. Inspect the ceph-nfs service:

    $ sudo pcs status | grep ceph-nfs

  3. Inspect the ceph-nfs systemd unit to confirm that it contains the Red Hat Ceph Storage 5 container image and tag: 

    $ cat /etc/systemd/system/ceph-nfs@.service | grep -i container_image

  4. Create a file called /home/stack/ganesha_update_extravars.yaml with the following content: 

    tripleo_cephadm_container_image: <ceph_image_name>
tripleo_cephadm_container_ns: <ceph_image_namespace>
tripleo_cephadm_container_tag: <ceph_image_tag>
    • Replace <ceph_image_name> with the name of the Red Hat Ceph Storage container image.
    • Replace <ceph_image_namespace> with the name of the Red Hat Ceph Storage container namespace.

    • Replace <ceph_image_tag> with the name of the Red Hat Ceph Storage container tag.

      For example, in a typical environment, this content would have the following values: 

      tripleo_cephadm_container_image: rhceph-6-rhel9
tripleo_cephadm_container_ns: undercloud-0.ctlplane.redhat.local:8787
tripleo_cephadm_container_tag: '6'
  5. Save the file.

  6. Run the ceph-update-ganesha.yml playbook and provide the ganesha_update_extravars.yaml playbook for additional command parameters: 

    ansible-playbook -i $HOME/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
    /usr/share/ansible/tripleo-playbooks/ceph-update-ganesha.yml \
     -e @$HOME/overcloud-deploy/<stack>/config-download/<stack>/global_vars.yaml \
     -e @$HOME/overcloud-deploy/<stack>/config-download/<stack>/cephadm/cephadm-extra-vars-heat.yml \
     -e @$HOME/ganesha_update_extravars.yaml
    • Replace <stack> with the name of the overcloud stack.

  7. Verify that the ceph-nfs service is running:

    $ sudo pcs status | grep ceph-nfs

  8. Verify that the ceph-nfs systemd unit contains the Red Hat Ceph Storage 6 container image and tag:

    $ cat /etc/systemd/system/ceph-nfs@.service | grep rhceph

When Red Hat Ceph Storage is upgraded from Release 4 to 5 to 6, more options must be set to configure the use of the Identity service than in older releases. For the procedure to set these options, see Configuring the Ceph Object Gateway to use Keystone authentication in the Object Gateway Guide.

Perform the following tasks if your Red Hat Ceph Storage cluster is external to your Red Hat OpenStack Platform deployment in your environment.

The container-image-prepare.yaml file contains the ContainerImagePrepare parameter and defines the Red Hat Ceph Storage containers. This file is used by the tripleo-container-image prepare command to define the rules for obtaining container images for the undercloud and overcloud. Update this file with the correct image version before updating your environment.

Procedure

  1. Locate your container preparation file. The default name of this file is containers-prepare-parameter.yaml.
  2. Edit the container preparation file.

  3. Locate the ceph_tag parameter. The current entry should be similar to the following example: 

    ceph_namespace: registry.redhat.io
ceph_image: rhceph-6-rhel9
ceph_tag: '5'

  4. Update the ceph_tag parameter for Red Hat Ceph Storage 6: 

    ceph_namespace: registry.redhat.io
ceph_image: rhceph-6-rhel9
ceph_tag: '6'

  5. Edit the containers-image-prepare.yaml file and replace the Red Hat Ceph monitoring stack container related parameters with the following content: 

    ceph_alertmanager_image: ose-prometheus-alertmanager
ceph_alertmanager_namespace: registry.redhat.io/openshift4
ceph_alertmanager_tag: v4.12
ceph_grafana_image: rhceph-6-dashboard-rhel9
ceph_grafana_namespace: registry.redhat.io/rhceph
ceph_grafana_tag: latest
ceph_node_exporter_image: ose-prometheus-node-exporter
ceph_node_exporter_namespace: registry.redhat.io/openshift4
ceph_node_exporter_tag: v4.12
ceph_prometheus_image: ose-prometheus
ceph_prometheus_namespace: registry.redhat.io/openshift4
ceph_prometheus_tag: v4.12
  6. Save the file.

16.2.2. Running the container image prepare
Copy link

Complete the container image preparation process by running the director container preparation command. This prepares all container image configurations for the overcloud and retrieves the latest Red Hat Ceph Storage 6 container image.

Note

If you are using Red Hat Satellite Server to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment, do not perform this procedure. Update Satellite to include the Red Hat Ceph Storage 6 container image and update your containers-prepare-parameter.yaml file to reference the URL of the container image that is hosted on the Satellite server.

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the stackrc undercloud credentials file: 

    $ source ~/stackrc

  3. Run the container preparation command:

    $ openstack tripleo container image prepare -e <container_preparation_file>

    • Replace <container_preparation_file> with the name of your file. The default file is containers-prepare-parameter.yaml.

  4. Verify that the new Red Hat Ceph Storage image is present in the undercloud registry:

    $ openstack tripleo container image list -f value | awk -F '//' '/ceph/ {print $2}'

  5. Verify that the new Red Hat Ceph Storage monitoring stack images are present in the undercloud registry:

    $ openstack tripleo container image list -f value | awk -F '//' '/dashboard|grafana|prometheus|alertmanager|node-exporter/ {print $2}'

When Red Hat Ceph Storage is upgraded from Release 4 to 5, NFS Ganesha is not adopted by the Orchestrator. This means it remains under director control and must be moved manually to Release 6.

Red Hat Ceph Storage 5 based NFS Ganesha with a Red Hat Ceph Storage 6 cluster is only supported during the upgrade period. Once you upgrade the Red Hat Ceph Storage cluster to 6, you must upgrade NFS Ganesha to use a Release 6 based container image.

Note

This procedure only applies to environments that are using the Shared File Systems service (manila) with CephFS NFS. Upgradng the Red Hat Ceph Storage container for NFS Ganesha is mandatory in these environments.

Procedure

  1. Log in to a Controller node.

  2. Inspect the ceph-nfs service:

    $ sudo pcs status | grep ceph-nfs

  3. Inspect the ceph-nfs systemd unit to confirm that it contains the Red Hat Ceph Storage 5 container image and tag: 

    $ cat /etc/systemd/system/ceph-nfs@.service | grep -i container_image

  4. Create a file called /home/stack/ganesha_update_extravars.yaml with the following content: 

    tripleo_cephadm_container_image: <ceph_image_name>
tripleo_cephadm_container_ns: <ceph_image_namespace>
tripleo_cephadm_container_tag: <ceph_image_tag>
    • Replace <ceph_image_name> with the name of the Red Hat Ceph Storage container image.
    • Replace <ceph_image_namespace> with the name of the Red Hat Ceph Storage container namespace.

    • Replace <ceph_image_tag> with the name of the Red Hat Ceph Storage container tag.

      For example, in a typical environment, this content would have the following values: 

      tripleo_cephadm_container_image: rhceph-6-rhel9
tripleo_cephadm_container_ns: undercloud-0.ctlplane.redhat.local:8787
tripleo_cephadm_container_tag: '6'
  5. Save the file.

  6. Run the ceph-update-ganesha.yml playbook and provide the ganesha_update_extravars.yaml playbook for additional command parameters: 

    ansible-playbook -i $HOME/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
    /usr/share/ansible/tripleo-playbooks/ceph-update-ganesha.yml \
     -e @$HOME/overcloud-deploy/<stack>/config-download/<stack>/global_vars.yaml \
     -e @$HOME/overcloud-deploy/<stack>/config-download/<stack>/cephadm/cephadm-extra-vars-heat.yml \
     -e @$HOME/ganesha_update_extravars.yaml
    • Replace <stack> with the name of the overcloud stack.

  7. Verify that the ceph-nfs service is running:

    $ sudo pcs status | grep ceph-nfs

  8. Verify that the ceph-nfs systemd unit contains the Red Hat Ceph Storage 6 container image and tag:

    $ cat /etc/systemd/system/ceph-nfs@.service | grep rhceph

When Red Hat Ceph Storage is upgraded from Release 4 to 5 to 6, more options must be set to configure the use of the Identity service than in older releases. For the procedure to set these options, see Configuring the Ceph Object Gateway to use Keystone authentication in the Object Gateway Guide.

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top