9.3. Upgrading to Red Hat Ceph Storage 5

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   • Replace <stack> with the name of your stack.
   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

4. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   重要

   The command in the previous step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific.

   $ sudo podman exec ceph-mon-$(hostname -f) ceph versions
   {
       "mon": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "mgr": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "osd": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180
       },
       "mds": {},
       "rgw": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "overall": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189
       }
   }

   注記

   The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container.

5. Create the ceph-admin user and distribute the appropriate keyrings:

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd

6. Update the packages on the Red Hat Ceph Storage nodes:

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_ansible_remote_tmp \
       --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     注記

     By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default).

7. Configure the Red Hat Ceph Storage nodes to use cephadm:

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     注記

     The adoption of cephadm can cause downtime in the RGW and Alertmanager services. For more information about these issues, see Restarting Red Hat Ceph Storage 5 services.

8. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   重要

   The command in the previous step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm. This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm.

   $ sudo cephadm shell -- ceph -s
     cluster:
       id:     f5a40da5-6d88-4315-9bb3-6b16df51d765
       health: HEALTH_WARN
               63 stray daemon(s) not managed by cephadm

9. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file:

   重要

   Do not include ceph-ansible environment or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml, in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy. For more information about replacing ceph-ansible environment and deployment files with cephadm files, see Implications of upgrading to Red Hat Ceph Storage 5.

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e <cephadm-file> \
     -e ~/containers-prepare-parameter.yaml

   where:

   <cephadm-file>

   • If you deployed RGW in a previous RHOSP version, or if you plan to deploy RGW, use environments/cephadm/cephadm.yaml.
   • If you plan to deploy RBD, use environments/cephadm/cephadm-rbd-only.yaml.

10. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

11. Save the file.

12. Run the upgrade preparation command:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

13. If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node:

    1. Log in to a Controller node:

       $ ssh cloud-admin@<controller_ip>

       • Replace <controller_ip> with the IP address of the Controller node.

    2. Retrieve a cephadm shell from the Controller node:

       Example

       [cloud-admin@controller-0 ~]$ sudo cephadm shell

    3. In the cephadm shell, create a temporary hci.conf file:

       Example

       [ceph: root@edpm-controller-0 /]# cat <<EOF > hci.conf
       [osd]
       osd_memory_target_autotune = true
       osd_numa_auto_affinity = true
       [mgr]
       mgr/cephadm/autotune_memory_target_ratio = 0.2
       EOF
       …​

    4. Apply the configuration:

       Example

       [ceph: root@edpm-controller-0 /]# ceph config assimilate-conf -i hci.conf

       For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure.