Chapter 6. Upgrading an overcloud with director-deployed Ceph deployments

Procedure

1. Enable the Ceph 5 Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

2. Determine if ceph-ansible is installed:

   [stack@director ~]$ sudo rpm -q ceph-ansible

3. Install or update the ceph-ansible package.

   1. If ceph-ansible is not installed, install the ceph-ansible package:

      [stack@director ~]$ sudo dnf install -y ceph-ansible

   2. If ceph-ansible is installed, update the ceph-ansible package to the latest version:

      [stack@director ~]$ sudo dnf update -y ceph-ansible

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   • Replace <stack> with the name of your stack.
   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

4. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific.

   $ sudo podman exec ceph-mon-$(hostname -f) ceph versions
   {
       "mon": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "mgr": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "osd": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180
       },
       "mds": {},
       "rgw": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "overall": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189
       }
   }

   Note

   The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container.

5. Create the ceph-admin user and distribute the appropriate keyrings:

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd

6. Update the packages on the Red Hat Ceph Storage nodes:

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_ansible_remote_tmp \
       --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default).

7. Configure the Red Hat Ceph Storage nodes to use cephadm:

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

8. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm. This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm.

   $ sudo cephadm shell -- ceph -s
     cluster:
       id:     f5a40da5-6d88-4315-9bb3-6b16df51d765
       health: HEALTH_WARN
               63 stray daemon(s) not managed by cephadm

9. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file.

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \
     -e ~/containers-prepare-parameter.yaml

   Note

   This example uses the environments/cephadm/cephadm-rbd-only.yaml file because RGW is not deployed. If you plan to deploy RGW, use environments/cephadm/cephadm.yaml after you finish upgrading your RHOSP environment, and then run a stack update.

10. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

11. Save the file.

12. Run the upgrade preparation command:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

13. If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node:

    1. Log in to a Controller node:

       $ ssh cloud-admin@<controller_ip>

       • Replace <controller_ip> with the IP address of the Controller node.

    2. Retrieve a cephadm shell from the Controller node:

       Example

       [cloud-admin@controller-0 ~]$ sudo cephadm shell

    3. In the cephadm shell, create a temporary hci.conf file:

       Example

       [ceph: root@edpm-controller-0 /]# cat <<EOF > hci.conf
       [osd]
       osd_memory_target_autotune = true
       osd_numa_auto_affinity = true
       [mgr]
       mgr/cephadm/autotune_memory_target_ratio = 0.2
       EOF
       …​

    4. Apply the configuration:

       Example

       [ceph: root@edpm-controller-0 /]# ceph config assimilate-conf -i hci.conf

       For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure.