Chapter 5. Rebooting the overcloud

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

   Copy to Clipboard Toggle word wrap

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

      Copy to Clipboard Toggle word wrap

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

      Copy to Clipboard Toggle word wrap

   3. If the node uses containerized services, check that all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

      Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you set the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name>

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log into the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and re-enable cluster rebalancing:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap
   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you unset the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name>

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

   Identify the UUID of the Compute node that you want to reboot.

3. From the undercloud, select a Compute node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

   Copy to Clipboard Toggle word wrap

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host <hostname> --all-projects

   Copy to Clipboard Toggle word wrap
5. If you decide not to migrate instances, skip to this step.

6. If you decide to migrate the instances to another Compute node, use one of the following commands:

   • Migrate the instance to a different host:

     (overcloud) $ openstack server migrate <instance_id> --live <target_host> --wait

     Copy to Clipboard Toggle word wrap

   • Let nova-scheduler automatically select the target host:

     (overcloud) $ nova live-migration <instance_id>

     Copy to Clipboard Toggle word wrap

   • Live migrate all instances at once:

     $ nova host-evacuate-live <hostname>

     Copy to Clipboard Toggle word wrap
     Note

     The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm that the migration was successful:

   (overcloud) $ openstack server list --host <hostname> --all-projects

   Copy to Clipboard Toggle word wrap
9. Continue to migrate instances until none remain on the chosen Compute node.

10. Log in to the Compute node and reboot the node:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

    Copy to Clipboard Toggle word wrap
11. Wait until the node boots.

12. Re-enable the Compute node:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set <hostname>  nova-compute --enable

    Copy to Clipboard Toggle word wrap

13. Check that the Compute node is enabled:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap