Chapter 14. Rebooting nodes

Procedure

1. Log into the undercloud as the stack user.

2. Reboot the undercloud:

   Copy to Clipboard Toggle word wrap

   $ sudo reboot

3. Wait until the node boots.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   Copy to Clipboard Toggle word wrap

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   Copy to Clipboard Toggle word wrap

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check the node has rejoined the cluster:

      Copy to Clipboard Toggle word wrap

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check all services are enabled:

      Copy to Clipboard Toggle word wrap

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check all containers on the node are active:

      Copy to Clipboard Toggle word wrap

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log into a Ceph MON node.

2. Reboot the node:

   Copy to Clipboard Toggle word wrap

   $ sudo reboot

3. Wait until the node boots and rejoins the MON cluster.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   Copy to Clipboard Toggle word wrap

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

2. Select the first Ceph Storage node to reboot and log into the node.

3. Reboot the node:

   Copy to Clipboard Toggle word wrap

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   Copy to Clipboard Toggle word wrap

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Check the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   Copy to Clipboard Toggle word wrap

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   Copy to Clipboard Toggle word wrap

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   Copy to Clipboard Toggle word wrap

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node you want to reboot.

3. From the undercloud, select a Compute Node. Disable the node:

   Copy to Clipboard Toggle word wrap

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

4. List all instances on the Compute node:

   Copy to Clipboard Toggle word wrap

   (overcloud) $ openstack server list --host [hostname] --all-projects

5. If you decided not to migrate instances, skip to this step.

6. If you decided to migrate the instances to another Compute node, use one of the following commands:

   1. Migrate the instance to a different host:

      Copy to Clipboard Toggle word wrap

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      Copy to Clipboard Toggle word wrap

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      Copy to Clipboard Toggle word wrap

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm the migration was successful:

   Copy to Clipboard Toggle word wrap

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Continue migrating instances until none remain on the chosen Compute Node.

10. Log in to the Compute Node. Reboot the node:

    Copy to Clipboard Toggle word wrap

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

11. Wait until the node boots.

12. Enable the Compute Node again:

    Copy to Clipboard Toggle word wrap

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

13. Check whether the Compute node is enabled:

    Copy to Clipboard Toggle word wrap

    (overcloud) $ openstack compute service list