Chapter 6. Upgrading an overcloud with director-deployed Ceph deployments

Procedure

1. Enable the Ceph 5 Tools repository:

   Copy to Clipboard Toggle word wrap

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   Copy to Clipboard Toggle word wrap

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to the undercloud node as the stack user.

2. Download the Red Hat Ceph Storage container image from the Satellite Server:

   Copy to Clipboard Toggle word wrap

   $ sudo podman pull <ceph_image_file>

   • Replace <ceph_image_file> with the Red Hat Ceph Storage container image file hosted on the Satellite Server. The following is an example of this command:

     Copy to Clipboard Toggle word wrap

     $ sudo podman pull satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   Copy to Clipboard Toggle word wrap

   $ source ~/stackrc

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   Copy to Clipboard Toggle word wrap

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   • Replace <stack> with the name of your stack.
   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

4. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific.

   Copy to Clipboard Toggle word wrap

   $ sudo podman exec ceph-mon-$(hostname -f) ceph versions
   {
       "mon": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "mgr": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "osd": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180
       },
       "mds": {},
       "rgw": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "overall": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189
       }
   }

   Note

   The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container.

5. Create the ceph-admin user and distribute the appropriate keyrings:

   Copy to Clipboard Toggle word wrap

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd

6. Update the packages on the Red Hat Ceph Storage nodes:

   Copy to Clipboard Toggle word wrap

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_ansible_remote_tmp \
       --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default).

7. Configure the Red Hat Ceph Storage nodes to use cephadm:

   Copy to Clipboard Toggle word wrap

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     The adoption of cephadm can cause downtime in the RGW and Alertmanager services. For more information about these issues, see Restarting Red Hat Ceph Storage 5 services.

8. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm. This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm.

   Copy to Clipboard Toggle word wrap

   $ sudo cephadm shell -- ceph -s
     cluster:
       id:     f5a40da5-6d88-4315-9bb3-6b16df51d765
       health: HEALTH_WARN
               63 stray daemon(s) not managed by cephadm

9. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file:

   Important

   Do not include ceph-ansible environment or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml, in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy. For more information about replacing ceph-ansible environment and deployment files with cephadm files, see Implications of upgrading to Red Hat Ceph Storage 5.

   Copy to Clipboard Toggle word wrap

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \
     -e ~/containers-prepare-parameter.yaml

   Note

   This example uses the environments/cephadm/cephadm-rbd-only.yaml file because RGW is not deployed. If you plan to deploy RGW, use environments/cephadm/cephadm.yaml after you finish upgrading your RHOSP environment, and then run a stack update.

10. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

    Copy to Clipboard Toggle word wrap

    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

11. Save the file.

12. Run the upgrade preparation command:

    Copy to Clipboard Toggle word wrap

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

13. If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node:

    1. Log in to a Controller node:

       Copy to Clipboard Toggle word wrap

       $ ssh cloud-admin@<controller_ip>

       • Replace <controller_ip> with the IP address of the Controller node.

    2. Retrieve a cephadm shell from the Controller node:

       Example

       Copy to Clipboard Toggle word wrap

       [cloud-admin@controller-0 ~]$ sudo cephadm shell

    3. In the cephadm shell, create a temporary hci.conf file:

       Example

       Copy to Clipboard Toggle word wrap

       [ceph: root@edpm-controller-0 /]# cat <<EOF > hci.conf
       [osd]
       osd_memory_target_autotune = true
       osd_numa_auto_affinity = true
       [mgr]
       mgr/cephadm/autotune_memory_target_ratio = 0.2
       EOF
       …​

    4. Apply the configuration:

       Example

       Copy to Clipboard Toggle word wrap

       [ceph: root@edpm-controller-0 /]# ceph config assimilate-conf -i hci.conf

       For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure.