Chapter 9. Upgrading an overcloud with director-deployed Ceph deployments

Procedure

1. Enable the Ceph 5 Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.

2. Download the Red Hat Ceph Storage container image from the Satellite Server:

   $ sudo podman pull <ceph_image_file>

   Copy to Clipboard Toggle word wrap

   • Replace <ceph_image_file> with the Red Hat Ceph Storage container image file hosted on the Satellite Server. The following is an example of this command:

     $ sudo podman pull satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest

     Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   Copy to Clipboard Toggle word wrap
   • Replace <stack> with the name of your stack.
   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

4. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific.

   $ sudo podman exec ceph-mon-$(hostname -f) ceph versions
   {
       "mon": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "mgr": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "osd": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180
       },
       "mds": {},
       "rgw": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "overall": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189
       }
   }

   Copy to Clipboard Toggle word wrap
   Note

   The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container.

5. Create the ceph-admin user and distribute the appropriate keyrings:

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd

   Copy to Clipboard Toggle word wrap

6. Update the packages on the Red Hat Ceph Storage nodes:

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_ansible_remote_tmp \
       --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   Copy to Clipboard Toggle word wrap

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default).

7. Configure the Red Hat Ceph Storage nodes to use cephadm:

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

   Copy to Clipboard Toggle word wrap

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     The adoption of cephadm can cause downtime in the RGW and Alertmanager services. For more information about these issues, see Restarting Red Hat Ceph Storage 5 services.

8. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm. This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm.

   $ sudo cephadm shell -- ceph -s
     cluster:
       id:     f5a40da5-6d88-4315-9bb3-6b16df51d765
       health: HEALTH_WARN
               63 stray daemon(s) not managed by cephadm

   Copy to Clipboard Toggle word wrap

9. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file:

   Important

   Do not include ceph-ansible environment or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml, in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy. For more information about replacing ceph-ansible environment and deployment files with cephadm files, see Implications of upgrading to Red Hat Ceph Storage 5.

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e <cephadm-file> \
     -e ~/containers-prepare-parameter.yaml

   Copy to Clipboard Toggle word wrap

   where:

   <cephadm-file>

   • If you deployed RGW in a previous RHOSP version, or if you plan to deploy RGW, use environments/cephadm/cephadm.yaml.
   • If you plan to deploy RBD, use environments/cephadm/cephadm-rbd-only.yaml.

10. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

    Copy to Clipboard Toggle word wrap
11. Save the file.

12. Run the upgrade preparation command:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

    Copy to Clipboard Toggle word wrap

13. If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node:

    1. Log in to a Controller node:

       $ ssh cloud-admin@<controller_ip>

       Copy to Clipboard Toggle word wrap
       • Replace <controller_ip> with the IP address of the Controller node.

    2. Retrieve a cephadm shell from the Controller node:

       Example

       [cloud-admin@controller-0 ~]$ sudo cephadm shell

       Copy to Clipboard Toggle word wrap

    3. In the cephadm shell, create a temporary hci.conf file:

       Example

       [ceph: root@edpm-controller-0 /]# cat <<EOF > hci.conf
       [osd]
       osd_memory_target_autotune = true
       osd_numa_auto_affinity = true
       [mgr]
       mgr/cephadm/autotune_memory_target_ratio = 0.2
       EOF
       …​

       Copy to Clipboard Toggle word wrap

    4. Apply the configuration:

       Example

       [ceph: root@edpm-controller-0 /]# ceph config assimilate-conf -i hci.conf

       Copy to Clipboard Toggle word wrap

       For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure.