Chapter 2. Upgrading a Red Hat Ceph Storage cluster running Red Hat Enterprise Linux 8 from RHCS 4 to RHCS 5

Procedure

1. Enable the Ceph and Ansible repositories on the Ansible administration node:

   Example

   [root@admin ceph-ansible]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

2. Update Ansible:

   Example

   [root@admin ceph-ansible]# dnf update ansible ceph-ansible

3. If the storage cluster you want to upgrade contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to create a denylist for clients:

   Syntax

   ceph auth caps client.ID mon 'profile rbd' osd 'profile rbd pool=POOL_NAME_1, profile rbd pool=POOL_NAME_2'

4. If the storage cluster was originally installed using Cockpit, create a symbolic link in the /usr/share/ceph-ansible directory to the inventory file where Cockpit created it, at /usr/share/ansible-runner-service/inventory/hosts:

   1. Change to the /usr/share/ceph-ansible directory:

      # cd /usr/share/ceph-ansible

   2. Create the symbolic link:

      # ln -s /usr/share/ansible-runner-service/inventory/hosts hosts

5. To upgrade the cluster using ceph-ansible, create the symbolic link in the etc/ansible/hosts directory to the hosts inventory file:

   # ln -s /etc/ansible/hosts hosts

6. If the storage cluster was originally installed using Cockpit, copy the Cockpit-generated SSH keys to the Ansible user’s ~/.ssh directory:

   1. Copy the keys:

      Syntax

      cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      cp /usr/share/ansible-runner-service/env/ssh_key /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Replace ANSIBLE_USERNAME with the user name for Ansible. The usual default user name is admin.

      Example

      # cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/admin/.ssh/id_rsa.pub
      # cp /usr/share/ansible-runner-service/env/ssh_key /home/admin/.ssh/id_rsa

   2. Set the appropriate owner, group, and permissions on the key files:

      Syntax

      # chown ANSIBLE_USERNAME:ANSIBLE_USERNAME /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chown ANSIBLE_USERNAME:ANSIBLE_USERNAME /home/ANSIBLE_USERNAME/.ssh/id_rsa
      # chmod 644 /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chmod 600 /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Replace ANSIBLE_USERNAME with the username for Ansible. The usual default user name is admin.

      Example

      # chown admin:admin /home/admin/.ssh/id_rsa.pub
      # chown admin:admin /home/admin/.ssh/id_rsa
      # chmod 644 /home/admin/.ssh/id_rsa.pub
      # chmod 600 /home/admin/.ssh/id_rsa

Procedure

1. Make a backup copy of the /etc/ceph and /var/lib/ceph folders.
2. Make a backup copy of the ceph.client.admin.keyring file.
3. Make backup copies of the ceph.conf files from each node.
4. Make backup copies of the /etc/ganesha/ folder on each node.
5. If the storage cluster has RBD mirroring defined, then make backup copies of the /etc/ceph folder and the group_vars/rbdmirrors.yml file.

Procedure

1. If you are running a multisite setup, set rgw_multisite: false in all.yml.

2. Ensure the group_vars/all.yml has the following default values for the configuration parameters:

   ceph_docker_image_tag: "latest"
   ceph_docker_registry: "registry.redhat.io"
   ceph_docker_image: rhceph/rhceph-4-rhel8
   containerized_deployment: true

   Note

   These values differ if you use a local registry and a custom image name.

3. Optional: For two-way RBD mirroring configured using the command-line interface in a bare-metal storage cluster, the cluster does not migrate RBD mirroring. For such a configuration, follow the below steps before migrating the non-containerized storage cluster to a containerized deployment:

   1. Create a user on the Ceph client node:

      Syntax

      ceph auth get client.PRIMARY_CLUSTER_NAME -o /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring

      Example

      [root@rbd-client-site-a ~]# ceph auth get client.rbd-mirror.site-a -o /etc/ceph/ceph.client.rbd-mirror.site-a.keyring

   2. Change the username in the auth file in /etc/ceph directory:

      Example

      [client.rbd-mirror.rbd-client-site-a]
          key = AQCbKbVg+E7POBAA7COSZCodvOrg2LWIFc9+3g==
          caps mds = "allow *"
          caps mgr = "allow *"
          caps mon = "allow *"
          caps osd = "allow *"

   3. Import the auth file to add relevant permissions:

      Syntax

      ceph auth import -i PATH_TO_KEYRING

      Example

      [root@rbd-client-site-a ~]# ceph auth import -i /etc/ceph/ceph.client.rbd-mirror.rbd-client-site-a.keyring

   4. Check the service name of the RBD mirror node:

      Example

      [root@rbd-client-site-a ~]# systemctl list-units --all
      
      systemctl stop ceph-rbd-mirror@rbd-client-site-a.service
      systemctl disable ceph-rbd-mirror@rbd-client-site-a.service
      systemctl reset-failed ceph-rbd-mirror@rbd-client-site-a.service
      systemctl start ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service
      systemctl enable ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service
      systemctl status ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service

   5. Add the rbd-mirror node to the /etc/ansible/hosts file:

      Example

      [rbdmirrors]
      ceph.client.rbd-mirror.rbd-client-site-a

4. If you are using daemons that are not containerized, convert them to containerized format:

   Syntax

   ansible-playbook -vvvv -i INVENTORY_FILE infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

   The -vvvv option collects verbose logs of the conversion process.

   Example

   [ceph-admin@admin ceph-ansible]$ ansible-playbook -vvvv -i hosts infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

5. Once the playbook completes successfully, edit the value of rgw_multisite: true in the all.yml file and ensure the value of containerized_deployment is true.

   Note

   Ensure to remove the ceph-iscsi, libtcmu, and tcmu-runner packages from the admin node.

Procedure

1. Enable the Ceph and Ansible repositories on the Ansible administration node:

   Red Hat Enterprise Linux 8

   subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

   Red Hat Enterprise Linux 9

   subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

2. On the Ansible administration node, ensure that the latest versions of the ansible and ceph-ansible packages are installed.

   Syntax

   dnf update ansible ceph-ansible

3. Navigate to the /usr/share/ceph-ansible/ directory:

   Example

   [root@admin ~]# cd /usr/share/ceph-ansible

4. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, make copies of the group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/osds.yml, and group_vars/clients.yml respectively.

   Example

   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml
   [root@admin ceph-ansible]# cp group_vars/rgws.yml.sample group_vars/rgws.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

5. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, edit the group_vars/all.yml file to add Red Hat Ceph Storage 5 details.

6. Once you have done the above two steps, copy the settings from the old yaml files to the new yaml files. Do not change the values of ceph_rhcs_version, ceph_docker_image, and grafana_container_image as the values for these configuration parameters are for Red Hat Ceph Storage 5. This ensures that all the settings related to your cluster are present in the current yaml file.

   Example

   fetch_directory: ~/ceph-ansible-keys
   monitor_interface: eth0
   public_network: 192.168.0.0/24
   ceph_docker_registry_auth: true
   ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
   ceph_docker_registry_password: TOKEN
   dashboard_admin_user: DASHBOARD_ADMIN_USERNAME
   dashboard_admin_password: DASHBOARD_ADMIN_PASSWORD
   grafana_admin_user: GRAFANA_ADMIN_USER
   grafana_admin_password: GRAFANA_ADMIN_PASSWORD
   radosgw_interface: eth0
   ceph_docker_image: "rhceph/rhceph-5-rhel8"
   ceph_docker_image_tag: "latest"
   ceph_docker_registry: "registry.redhat.io"
   node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
   grafana_container_image: registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8:5
   prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
   alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

   Note

   Ensure the Red Hat Ceph Storage 5 container images are set to the default values.

7. Edit the group_vars/osds.yml file. Add and set the following options:

   Syntax

   nb_retry_wait_osd_up: 50
   delay_wait_osd_up: 30

8. Open the group_vars/all.yml file and verify the following values are present from the old all.yml file.

   1. The fetch_directory option is set with the same value from the old all.yml file:

      Syntax

      fetch_directory: FULL_DIRECTORY_PATH

      Replace FULL_DIRECTORY_PATH with a writable location, such as the Ansible user’s home directory.

   2. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface option:

      radosgw_interface: INTERFACE

      Replace INTERFACE with the interface to which the Ceph Object Gateway nodes listen.

   3. If your current setup has SSL certificates configured, edit the following:

      Syntax

      radosgw_frontend_ssl_certificate: /etc/pki/ca-trust/extracted/CERTIFICATE_NAME
      radosgw_frontend_port: 443

   4. Uncomment the upgrade_ceph_packages option and set it to True:

      Syntax

      upgrade_ceph_packages: True

   5. If the storage cluster has more than one Ceph Object Gateway instance per node, then uncomment the radosgw_num_instances setting and set it to the number of instances per node in the cluster:

      Syntax

      radosgw_num_instances : NUMBER_OF_INSTANCES_PER_NODE

      Example

      radosgw_num_instances : 2

   6. If the storage cluster has Ceph Object Gateway multi-site defined, check the multisite settings in all.yml to make sure that they contain the same values as they did in the old all.yml file.

9. If the buckets are created or have the num_shards = 0, manually reshard the buckets, before planning an upgrade to Red Hat Ceph Storage 5.3:

   Warning

   Upgrade to Red Hat Ceph Storage 5.3 from older releases when bucket_index_max_shards is 0 can result in the loss of the Ceph Object Gateway bucket’s metadata leading to the bucket’s unavailability while trying to access it. Hence, ensure bucket_index_max_shards is set to 11 shards. If not, modify this configuration at the zonegroup level.

   Syntax

   radosgw-admin bucket reshard --num-shards 11 --bucket BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin bucket reshard --num-shards 11 --bucket mybucket

10. Log in as ansible-user on the Ansible administration node.

11. Use the --extra-vars option to update the infrastructure-playbooks/rolling_update.yml playbook and to change the health_osd_check_retries and health_osd_check_delay values to 50 and 30, respectively:

    Example

    [root@admin ceph-ansible]# ansible-playbook -i hosts infrastructure-playbooks/rolling_update.yml --extra-vars "health_osd_check_retries=50 health_osd_check_delay=30"

    For each OSD node, these values cause ceph-ansible to check the storage cluster health every 30 seconds, up to 50 times. This means that ceph-ansible waits up to 25 minutes for each OSD.

    Adjust the health_osd_check_retries option value up or down, based on the used storage capacity of the storage cluster. For example, if you are using 218 TB out of 436 TB, or 50% of the storage capacity, then set the health_osd_check_retries option to 50.

    /etc/ansible/hosts is the default location for the Ansible inventory file.

12. Run the rolling_update.yml playbook to convert the storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5:

    Syntax

    ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i INVENTORY_FILE

    The -vvvv option collects verbose logs of the upgrade process.

    Example

    [ceph-admin@admin ceph-ansible]$ ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i hosts

    Important

    Using the --limit Ansible option with the rolling_update.yml playbook is not supported.

13. Review the Ansible playbook log output to verify the status of the upgrade.

Verification

1. List all running containers:

   Example

   [root@mon ~]# podman ps

2. Check the health status of the cluster. Replace MONITOR_ID with the name of the Ceph Monitor container found in the previous step:

   Syntax

   podman exec ceph-mon-MONITOR_ID ceph -s

   Example

   [root@mon ~]# podman exec ceph-mon-mon01 ceph -s

3. Verify the Ceph cluster daemon versions to confirm the upgrade of all daemons. Replace MONITOR_ID with the name of the Ceph Monitor container found in the previous step:

   Syntax

   podman exec ceph-mon-MONITOR_ID ceph --cluster ceph versions

   Example

   [root@mon ~]# podman exec ceph-mon-mon01 ceph --cluster ceph versions

Procedure

1. Log in to the ceph-ansible node and change directory to /usr/share/ceph-ansible.

2. Edit the all.yml file.

   Syntax

   ceph_origin: custom/rhcs
   ceph_custom_repositories:
     - name: NAME
       state: present
       description: DESCRIPTION
       gpgcheck: 'no'
       baseurl: BASE_URL
       file: FILE_NAME
       priority: '2'
       enabled: 1

   Example

   ceph_origin: custom
   ceph_custom_repositories:
     - name: ceph_custom
       state: present
       description: Ceph custom repo
       gpgcheck: 'no'
       baseurl: https://example.ceph.redhat.com
       file: cephbuild
       priority: '2'
       enabled: 1
     - name: ceph_custom_1
       state: present
       description: Ceph custom repo 1
       gpgcheck: 'no'
       baseurl: https://example.ceph.redhat.com
       file: cephbuild_1
       priority: '2'
       enabled: 1

3. Run the cephadm-adopt playbook:

   Syntax

   ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i INVENTORY_FILE

   Example

   [ceph-admin@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i hosts

4. Set the minimum compat client parameter to luminous:

   Example

   [ceph: root@node0 /]# ceph osd set-require-min-compat-client luminous

5. Run the following command to enable applications to run on the NFS-Ganesha pool. POOL_NAME is nfs-ganesha, and APPLICATION_NAME is the name of the application you want to enable, such as cephfs, rbd, or rgw.

   Syntax

   ceph osd pool application enable POOL_NAME APPLICATION_NAME

   Example

   [ceph: root@node0 /]# ceph osd pool application enable nfs-ganesha rgw

   Important

   The cephadm-adopt playbook does not bring up rbd-mirroring after migrating the storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5.

   To work around this issue, add the peers manually:

   Syntax

   rbd mirror pool peer add POOL_NAME CLIENT_NAME@CLUSTER_NAME

   Example

   [ceph: root@node0 /]# rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b

6. Remove Grafana after upgrade:

   1. Log in to the Cephadm shell:

      Example

      [root@host01 ~]# cephadm shell

   2. Fetch the name of Grafana in your storage cluster:

      Example

      [ceph: root@host01 /]# ceph orch ps --daemon_type grafana

   3. Remove Grafana:

      Syntax

      ceph orch daemon rm GRAFANA_DAEMON_NAME

      Example

      [ceph: root@host01 /]# ceph orch daemon rm grafana.host01
      
      Removed grafana.host01 from host 'host01'

   4. Wait a few minutes and check the latest log:

      Example

      [ceph: root@host01 /]# ceph log last cephadm

      cephadm redeploys the Grafana service and the daemon.

Procedure

1. Uninstall ansible and the older ceph-ansible packages:

   Syntax

   dnf remove ansible ceph-ansible

2. Disable Ansible repository and enable Ceph repository on the Ansible administration node:

   Red Hat Enterprise Linux 8

   [root@admin ~]# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.9-for-rhel-8-x86_64-rpms

   Red Hat Enterprise Linux 9

   [root@admin ~]# subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms --enable=rhel-9-for-x86_64-appstream-rpms --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms --disable=ansible-2.9-for-rhel-9-x86_64-rpms

3. Install the cephadm-ansible package, which installs the ansible-core as a dependency:

   Syntax

   dnf install cephadm-ansible