Chapter 1. Upgrading a Red Hat Ceph Storage cluster from RHCS 4 to RHCS 5

Procedure

1. Enable the Ceph and Ansible repositories on the Ansible administration node:

   Example

   [root@admin ceph-ansible]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

2. Update Ansible:

   Example

   [root@admin ceph-ansible]# dnf update ansible ceph-ansible

   Copy to Clipboard Toggle word wrap

3. If the storage cluster you want to upgrade contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to create a denylist for clients:

   Syntax

   ceph auth caps client.ID mon 'profile rbd' osd 'profile rbd pool=POOL_NAME_1, profile rbd pool=POOL_NAME_2'

   Copy to Clipboard Toggle word wrap

4. If the storage cluster was originally installed using Cockpit, create a symbolic link in the /usr/share/ceph-ansible directory to the inventory file where Cockpit created it, at /usr/share/ansible-runner-service/inventory/hosts:

   1. Change to the /usr/share/ceph-ansible directory:

      # cd /usr/share/ceph-ansible

      Copy to Clipboard Toggle word wrap

   2. Create the symbolic link:

      # ln -s /usr/share/ansible-runner-service/inventory/hosts hosts

      Copy to Clipboard Toggle word wrap

5. To upgrade the cluster using ceph-ansible, create the symbolic link in the etc/ansible/hosts directory to the hosts inventory file:

   # ln -s /etc/ansible/hosts hosts

   Copy to Clipboard Toggle word wrap

6. If the storage cluster was originally installed using Cockpit, copy the Cockpit-generated SSH keys to the Ansible user’s ~/.ssh directory:

   1. Copy the keys:

      Syntax

      cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      cp /usr/share/ansible-runner-service/env/ssh_key /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Copy to Clipboard Toggle word wrap

      Replace ANSIBLE_USERNAME with the user name for Ansible. The usual default user name is admin.

      Example

      # cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/admin/.ssh/id_rsa.pub
      # cp /usr/share/ansible-runner-service/env/ssh_key /home/admin/.ssh/id_rsa

      Copy to Clipboard Toggle word wrap

   2. Set the appropriate owner, group, and permissions on the key files:

      Syntax

      # chown ANSIBLE_USERNAME:ANSIBLE_USERNAME /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chown ANSIBLE_USERNAME:ANSIBLE_USERNAME /home/ANSIBLE_USERNAME/.ssh/id_rsa
      # chmod 644 /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chmod 600 /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Copy to Clipboard Toggle word wrap

      Replace ANSIBLE_USERNAME with the username for Ansible. The usual default user name is admin.

      Example

      # chown admin:admin /home/admin/.ssh/id_rsa.pub
      # chown admin:admin /home/admin/.ssh/id_rsa
      # chmod 644 /home/admin/.ssh/id_rsa.pub
      # chmod 600 /home/admin/.ssh/id_rsa

      Copy to Clipboard Toggle word wrap

Procedure

1. Make a backup copy of the /etc/ceph and /var/lib/ceph folders.
2. Make a backup copy of the ceph.client.admin.keyring file.
3. Make backup copies of the ceph.conf files from each node.
4. Make backup copies of the /etc/ganesha/ folder on each node.
5. If the storage cluster has RBD mirroring defined, then make backup copies of the /etc/ceph folder and the group_vars/rbdmirrors.yml file.

Procedure

1. If you are running a multisite setup, set rgw_multisite: false in all.yml.

2. Ensure the group_vars/all.yml has the following default values for the configuration parameters:

   ceph_docker_image_tag: "latest"
   ceph_docker_registry: "registry.redhat.io"
   ceph_docker_image: rhceph/rhceph-4-rhel8
   containerized_deployment: true

   Copy to Clipboard Toggle word wrap
   Note

   These values differ if you use a local registry and a custom image name.

3. Optional: For two-way RBD mirroring configured using the command-line interface in a bare-metal storage cluster, the cluster does not migrate RBD mirroring. For such a configuration, follow the below steps before migrating the non-containerized storage cluster to a containerized deployment:

   1. Create a user on the Ceph client node:

      Syntax

      ceph auth get client.PRIMARY_CLUSTER_NAME -o /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring

      Copy to Clipboard Toggle word wrap

      Example

      [root@rbd-client-site-a ~]# ceph auth get client.rbd-mirror.site-a -o /etc/ceph/ceph.client.rbd-mirror.site-a.keyring

      Copy to Clipboard Toggle word wrap

   2. Change the username in the auth file in /etc/ceph directory:

      Example

      [client.rbd-mirror.rbd-client-site-a]
          key = AQCbKbVg+E7POBAA7COSZCodvOrg2LWIFc9+3g==
          caps mds = "allow *"
          caps mgr = "allow *"
          caps mon = "allow *"
          caps osd = "allow *"

      Copy to Clipboard Toggle word wrap

   3. Import the auth file to add relevant permissions:

      Syntax

      ceph auth import -i PATH_TO_KEYRING

      Copy to Clipboard Toggle word wrap

      Example

      [root@rbd-client-site-a ~]# ceph auth import -i /etc/ceph/ceph.client.rbd-mirror.rbd-client-site-a.keyring

      Copy to Clipboard Toggle word wrap

   4. Check the service name of the RBD mirror node:

      Example

      [root@rbd-client-site-a ~]# systemctl list-units --all
      
      systemctl stop ceph-rbd-mirror@rbd-client-site-a.service
      systemctl disable ceph-rbd-mirror@rbd-client-site-a.service
      systemctl reset-failed ceph-rbd-mirror@rbd-client-site-a.service
      systemctl start ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service
      systemctl enable ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service
      systemctl status ceph-rbd-mirror@rbd-mirror.rbd-client-site-a.service

      Copy to Clipboard Toggle word wrap

   5. Add the rbd-mirror node to the /etc/ansible/hosts file:

      Example

      [rbdmirrors]
      ceph.client.rbd-mirror.rbd-client-site-a

      Copy to Clipboard Toggle word wrap

4. If you are using daemons that are not containerized, convert them to containerized format:

   Syntax

   ansible-playbook -vvvv -i INVENTORY_FILE infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

   Copy to Clipboard Toggle word wrap

   The -vvvv option collects verbose logs of the conversion process.

   Example

   [ceph-admin@admin ceph-ansible]$ ansible-playbook -vvvv -i hosts infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

   Copy to Clipboard Toggle word wrap

5. Once the playbook completes successfully, edit the value of rgw_multisite: true in the all.yml file and ensure the value of containerized_deployment is true.

   Note

   Ensure to remove the ceph-iscsi, libtcmu, and tcmu-runner packages from the admin node.

Procedure

1. Use the docker-to-podman playbook to convert docker to podman:

   Example

   [ceph-admin@admin ceph-ansible]$ ansible-playbook -vvvv -i hosts infrastructure-playbooks/
   docker-to-podman.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. Restore the files that you backed up before the host OS upgrade to the host.
2. Restore the /etc/ceph folders and their contents to all of the hosts, including the ceph.client.admin.keyring and ceph.conf files.
3. Restore the /etc/ganesha/ folder to each node.

4. Check to make sure that the ownership for each of the backed-up files has not changed after the operating system upgrade. The file owner should be ceph. If the file owner has been changed to root, use the following command on each file to change the ownership back to ceph:

   Example

   [root@admin]# chown ceph: ceph.client.rbd-mirror.node1.keyring

   Copy to Clipboard Toggle word wrap

5. If you upgraded from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 and the storage cluster had RBD mirroring defined, restore the /etc/ceph folder from the backup copy.
6. Restore the group_vars/rbdmirrors.yml file that you backed up earlier.

7. Change the ownership of the folders on all nodes:

   Example

   [root@admin]# chown -R /etc/ceph
   [root@admin]# chown -R /var/log/ceph
   [root@admin]# chown -R /var/lib/ceph

   Copy to Clipboard Toggle word wrap

Procedure

1. Enable the Ceph and Ansible repositories on the Ansible administration node:

   Red Hat Enterprise Linux 8

   subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 9

   subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

   Copy to Clipboard Toggle word wrap

2. On the Ansible administration node, ensure that the latest versions of the ansible and ceph-ansible packages are installed.

   Syntax

   dnf update ansible ceph-ansible

   Copy to Clipboard Toggle word wrap

3. Navigate to the /usr/share/ceph-ansible/ directory:

   Example

   [root@admin ~]# cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

4. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, make copies of the group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/osds.yml, and group_vars/clients.yml respectively.

   Example

   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml
   [root@admin ceph-ansible]# cp group_vars/rgws.yml.sample group_vars/rgws.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

   Copy to Clipboard Toggle word wrap

5. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, edit the group_vars/all.yml file to add Red Hat Ceph Storage 5 details.

6. Once you have done the above two steps, copy the settings from the old yaml files to the new yaml files. Do not change the values of ceph_rhcs_version, ceph_docker_image, and grafana_container_image as the values for these configuration parameters are for Red Hat Ceph Storage 5. This ensures that all the settings related to your cluster are present in the current yaml file.

   Example

   fetch_directory: ~/ceph-ansible-keys
   monitor_interface: eth0
   public_network: 192.168.0.0/24
   ceph_docker_registry_auth: true
   ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
   ceph_docker_registry_password: TOKEN
   dashboard_admin_user: DASHBOARD_ADMIN_USERNAME
   dashboard_admin_password: DASHBOARD_ADMIN_PASSWORD
   grafana_admin_user: GRAFANA_ADMIN_USER
   grafana_admin_password: GRAFANA_ADMIN_PASSWORD
   radosgw_interface: eth0
   ceph_docker_image: "rhceph/rhceph-5-rhel8"
   ceph_docker_image_tag: "latest"
   ceph_docker_registry: "registry.redhat.io"
   node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
   grafana_container_image: registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8:5
   prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
   alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

   Copy to Clipboard Toggle word wrap

   Note

   Ensure the Red Hat Ceph Storage 5 container images are set to the default values.

7. Edit the group_vars/osds.yml file. Add and set the following options:

   Syntax

   nb_retry_wait_osd_up: 50
   delay_wait_osd_up: 30

   Copy to Clipboard Toggle word wrap

8. Open the group_vars/all.yml file and verify the following values are present from the old all.yml file.

   1. The fetch_directory option is set with the same value from the old all.yml file:

      Syntax

      fetch_directory: FULL_DIRECTORY_PATH

      Copy to Clipboard Toggle word wrap

      Replace FULL_DIRECTORY_PATH with a writable location, such as the Ansible user’s home directory.

   2. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface option:

      radosgw_interface: INTERFACE

      Copy to Clipboard Toggle word wrap

      Replace INTERFACE with the interface to which the Ceph Object Gateway nodes listen.

   3. If your current setup has SSL certificates configured, edit the following:

      Syntax

      radosgw_frontend_ssl_certificate: /etc/pki/ca-trust/extracted/CERTIFICATE_NAME
      radosgw_frontend_port: 443

      Copy to Clipboard Toggle word wrap

   4. Uncomment the upgrade_ceph_packages option and set it to True:

      Syntax

      upgrade_ceph_packages: True

      Copy to Clipboard Toggle word wrap

   5. If the storage cluster has more than one Ceph Object Gateway instance per node, then uncomment the radosgw_num_instances setting and set it to the number of instances per node in the cluster:

      Syntax

      radosgw_num_instances : NUMBER_OF_INSTANCES_PER_NODE

      Copy to Clipboard Toggle word wrap

      Example

      radosgw_num_instances : 2

      Copy to Clipboard Toggle word wrap

   6. If the storage cluster has Ceph Object Gateway multi-site defined, check the multisite settings in all.yml to make sure that they contain the same values as they did in the old all.yml file.

9. If the buckets are created or have the num_shards = 0, manually reshard the buckets, before planning an upgrade to Red Hat Ceph Storage 5.3:

   Warning

   Upgrade to Red Hat Ceph Storage 5.3 from older releases when bucket_index_max_shards is 0 can result in the loss of the Ceph Object Gateway bucket’s metadata leading to the bucket’s unavailability while trying to access it. Hence, ensure bucket_index_max_shards is set to 11 shards. If not, modify this configuration at the zonegroup level.

   Syntax

   radosgw-admin bucket reshard --num-shards 11 --bucket BUCKET_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# radosgw-admin bucket reshard --num-shards 11 --bucket mybucket

   Copy to Clipboard Toggle word wrap

10. Log in as ansible-user on the Ansible administration node.

11. Use the --extra-vars option to update the infrastructure-playbooks/rolling_update.yml playbook and to change the health_osd_check_retries and health_osd_check_delay values to 50 and 30, respectively:

    Example

    [root@admin ceph-ansible]# ansible-playbook -i hosts infrastructure-playbooks/rolling_update.yml --extra-vars "health_osd_check_retries=50 health_osd_check_delay=30"

    Copy to Clipboard Toggle word wrap

    For each OSD node, these values cause ceph-ansible to check the storage cluster health every 30 seconds, up to 50 times. This means that ceph-ansible waits up to 25 minutes for each OSD.

    Adjust the health_osd_check_retries option value up or down, based on the used storage capacity of the storage cluster. For example, if you are using 218 TB out of 436 TB, or 50% of the storage capacity, then set the health_osd_check_retries option to 50.

    /etc/ansible/hosts is the default location for the Ansible inventory file.

12. Run the rolling_update.yml playbook to convert the storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5:

    Syntax

    ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i INVENTORY_FILE

    Copy to Clipboard Toggle word wrap

    The -vvvv option collects verbose logs of the upgrade process.

    Example

    [ceph-admin@admin ceph-ansible]$ ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i hosts

    Copy to Clipboard Toggle word wrap

    Important

    Using the --limit Ansible option with the rolling_update.yml playbook is not supported.

13. Review the Ansible playbook log output to verify the status of the upgrade.

Verification

1. List all running containers:

   Example

   [root@mon ~]# podman ps

   Copy to Clipboard Toggle word wrap

2. Check the health status of the cluster. Replace MONITOR_ID with the name of the Ceph Monitor container found in the previous step:

   Syntax

   podman exec ceph-mon-MONITOR_ID ceph -s

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# podman exec ceph-mon-mon01 ceph -s

   Copy to Clipboard Toggle word wrap

3. Verify the Ceph cluster daemon versions to confirm the upgrade of all daemons. Replace MONITOR_ID with the name of the Ceph Monitor container found in the previous step:

   Syntax

   podman exec ceph-mon-MONITOR_ID ceph --cluster ceph versions

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# podman exec ceph-mon-mon01 ceph --cluster ceph versions

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the ceph-ansible node and change directory to /usr/share/ceph-ansible.

2. Edit the all.yml file.

   Syntax

   ceph_origin: custom/rhcs
   ceph_custom_repositories:
     - name: NAME
       state: present
       description: DESCRIPTION
       gpgcheck: 'no'
       baseurl: BASE_URL
       file: FILE_NAME
       priority: '2'
       enabled: 1

   Copy to Clipboard Toggle word wrap

   Example

   ceph_origin: custom
   ceph_custom_repositories:
     - name: ceph_custom
       state: present
       description: Ceph custom repo
       gpgcheck: 'no'
       baseurl: https://example.ceph.redhat.com
       file: cephbuild
       priority: '2'
       enabled: 1
     - name: ceph_custom_1
       state: present
       description: Ceph custom repo 1
       gpgcheck: 'no'
       baseurl: https://example.ceph.redhat.com
       file: cephbuild_1
       priority: '2'
       enabled: 1

   Copy to Clipboard Toggle word wrap

3. Run the cephadm-adopt playbook:

   Syntax

   ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i INVENTORY_FILE

   Copy to Clipboard Toggle word wrap

   Example

   [ceph-admin@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i hosts

   Copy to Clipboard Toggle word wrap

4. Set the minimum compat client parameter to luminous:

   Example

   [ceph: root@node0 /]# ceph osd set-require-min-compat-client luminous

   Copy to Clipboard Toggle word wrap

5. Run the following command to enable applications to run on the NFS-Ganesha pool. POOL_NAME is nfs-ganesha, and APPLICATION_NAME is the name of the application you want to enable, such as cephfs, rbd, or rgw.

   Syntax

   ceph osd pool application enable POOL_NAME APPLICATION_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@node0 /]# ceph osd pool application enable nfs-ganesha rgw

   Copy to Clipboard Toggle word wrap

   Important

   The cephadm-adopt playbook does not bring up rbd-mirroring after migrating the storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5.

   To work around this issue, add the peers manually:

   Syntax

   rbd mirror pool peer add POOL_NAME CLIENT_NAME@CLUSTER_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@node0 /]# rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b

   Copy to Clipboard Toggle word wrap

6. Remove Grafana after upgrade:

   1. Log in to the Cephadm shell:

      Example

      [root@host01 ~]# cephadm shell

      Copy to Clipboard Toggle word wrap

   2. Fetch the name of Grafana in your storage cluster:

      Example

      [ceph: root@host01 /]# ceph orch ps --daemon_type grafana

      Copy to Clipboard Toggle word wrap

   3. Remove Grafana:

      Syntax

      ceph orch daemon rm GRAFANA_DAEMON_NAME

      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph orch daemon rm grafana.host01
      
      Removed grafana.host01 from host 'host01'

      Copy to Clipboard Toggle word wrap

   4. Wait a few minutes and check the latest log:

      Example

      [ceph: root@host01 /]# ceph log last cephadm

      Copy to Clipboard Toggle word wrap

      cephadm redeploys the Grafana service and the daemon.

Procedure

1. Uninstall ansible and the older ceph-ansible packages:

   Syntax

   dnf remove ansible ceph-ansible

   Copy to Clipboard Toggle word wrap

2. Disable Ansible repository and enable Ceph repository on the Ansible administration node:

   Red Hat Enterprise Linux 8

   [root@admin ~]# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.9-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 9

   [root@admin ~]# subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms --enable=rhel-9-for-x86_64-appstream-rpms --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms --disable=ansible-2.9-for-rhel-9-x86_64-rpms

   Copy to Clipboard Toggle word wrap

3. Install the cephadm-ansible package, which installs the ansible-core as a dependency:

   Syntax

   dnf install cephadm-ansible

   Copy to Clipboard Toggle word wrap