Chapter 12. Rebooting Nodes

Procedure

1. Log into the undercloud as the stack user.

2. Reboot the undercloud:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
3. Wait until the node boots.

Procedure

1. Select a node to reboot. Log into the node and stop the cluster before rebooting:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

   Copy to Clipboard Toggle word wrap

2. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
3. Wait until the node boots.

4. Re-enable the cluster for the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster start

   Copy to Clipboard Toggle word wrap

5. Log into the node and check the services:

   1. If the node uses Pacemaker services, check the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

      Copy to Clipboard Toggle word wrap

   2. If the node uses Systemd services, check all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

      Copy to Clipboard Toggle word wrap

   3. If the node uses containerized services, check all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo docker ps

      Copy to Clipboard Toggle word wrap

Procedure

1. Log into a Ceph MON node.

2. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
3. Wait until the node boots and rejoins the MON cluster.

Procedure

1. Log into a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node to reboot and log into the node.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log into the node and check the cluster status:

   $ sudo ceph -s

   Copy to Clipboard Toggle word wrap

   Check the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo ceph osd unset noout
   $ sudo ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo ceph status

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

   Identify the UUID of the Compute node you want to reboot.

3. From the undercloud, select a Compute Node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

   Copy to Clipboard Toggle word wrap

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap

5. Use one of the following commands to migrate your instances:

   1. Migrate the instance to a different host:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

      Copy to Clipboard Toggle word wrap

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

      Copy to Clipboard Toggle word wrap

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Copy to Clipboard Toggle word wrap
      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

6. Wait until migration completes.

7. Confirm the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap
8. Continue migrating instances until none remain on the chosen Compute Node.

9. Log into the Compute Node. Reboot the node:

   [heat-admin@overcloud-compute-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
10. Wait until the node boots.

11. Enable the Compute Node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

    Copy to Clipboard Toggle word wrap

12. Check whether the Compute node is enabled:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap