Chapter 3. Performing maintenance on the undercloud and overcloud with Instance HA

Procedure

1. Log in to the undercloud as the stack user.

2. Source the credentials file for your overcloud:

   $ source ~/overcloudrc

   Copy to Clipboard

3. View running instances in the overcloud:

   $ openstack server list --all-projects

   Copy to Clipboard

4. Stop each instance in the overcloud:

   $ openstack server stop <INSTANCE>

   Copy to Clipboard

   Repeat this step for each instance until you stop all instances in the overcloud.

Procedure

1. Log in as the root user to an overcloud node that runs Pacemaker.

2. Disable the Pacemaker remote resource on each Compute node:

   1. Identify the Pacemaker Remote resource on Compute nodes:

      # pcs resource status

      Copy to Clipboard

      These resources use the ocf::pacemaker:remote agent and are usually named after the Compute node host format, such as overcloud-novacomputeiha-0.

   2. Disable each Pacemaker Remote resource. The following example shows how to disable the resource for overcloud-novacomputeiha-0:

      # pcs resource disable overcloud-novacomputeiha-0

      Copy to Clipboard

3. Disable the Compute node STONITH devices:

   1. Identify the Compute node STONITH devices:

      # pcs stonith status

      Copy to Clipboard

   2. Disable each Compute node STONITH device:

      # pcs stonith disable <STONITH_DEVICE>

      Copy to Clipboard

Procedure

1. Log in as the root user to a Compute node.

2. Shut down the node:

   # shutdown -h now

   Copy to Clipboard
3. Perform these steps for each Compute node until you shut down all Compute nodes.

Procedure

1. Log in as the root user to a Controller node.

2. Stop the Pacemaker cluster.

   # pcs cluster stop --all

   Copy to Clipboard

   This command stops the cluster on all nodes.

3. Wait until the Pacemaker services stop and check that the services stopped.

   1. Check the Pacemaker status:

      # pcs status

      Copy to Clipboard

   2. Check that no Pacemaker services are running in Podman:

      # podman ps --filter "name=.*-bundle.*"

      Copy to Clipboard

4. Stop the Red Hat OpenStack Platform services:

   # systemctl stop 'tripleo_*'

   Copy to Clipboard

5. Wait until the services stop and check that services are no longer running in Podman:

   # podman ps

   Copy to Clipboard

Procedure

1. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.

2. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard

   Ensure that the status is HEALTH_OK.

3. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags for the cluster. In the following example, the podman commands set these flags through a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set norecover
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set nobackfill
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set nodown
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set pause

   Copy to Clipboard

4. Shut down each Ceph Storage node:

   1. Log in as the root user to a Ceph Storage node.

   2. Shut down the node:

      # shutdown -h now

      Copy to Clipboard
   3. Perform these steps for each Ceph Storage node until you shut down all Ceph Storage nodes.

5. Shut down any standalone Ceph MON nodes:

   1. Log in as the root user to a standalone Ceph MON node.

   2. Shut down the node:

      # shutdown -h now

      Copy to Clipboard
   3. Perform these steps for each standalone Ceph MON node until you shut down all standalone Ceph MON nodes.

Procedure

1. Log in as the root user to a Controller node.

2. Shut down the node:

   # shutdown -h now

   Copy to Clipboard
3. Perform these steps for each Controller node until you shut down all Controller nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Shut down the undercloud:

   $ sudo shutdown -h now

   Copy to Clipboard

Verification

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard

3. Check the services on the undercloud:

   $ systemctl list-units 'tripleo_*'

   Copy to Clipboard

4. Validate the static inventory file named tripleo-ansible-inventory.yaml:

   $ validation run --group pre-introspection -i <inventory_file>

   Copy to Clipboard

   • Replace <inventory_file> with the name and location of the Ansible inventory file, for example, ~/tripleo-deploy/undercloud/tripleo-ansible-inventory.yaml.

     Note

     When you run a validation, the Reasons column in the output is limited to 79 characters. To view the validation result in full, view the validation log files.

5. Check that all services and containers are active and healthy:

   $ validation run --validation service-status --limit undercloud -i <inventory_file>

   Copy to Clipboard

Verification

1. Log in to each Controller node as the root user.

2. Check the services on the Controller node:

   $ systemctl -t service

   Copy to Clipboard

   Only non-Pacemaker based services are running.

3. Wait until the Pacemaker services start and check that the services started:

   $ pcs status

   Copy to Clipboard
   Note

   If your environment uses Instance HA, the Pacemaker resources do not start until you start the Compute nodes or perform a manual unfence operation with the pcs stonith confirm <compute_node> command. You must run this command on each Compute node that uses Instance HA.

Procedure

1. If your environment has standalone Ceph MON nodes, power on each Ceph MON node.
2. Power on each Ceph Storage node.
3. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.

4. Check the status of the cluster nodes. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard

   Ensure that each node is powered on and connected.

5. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags for the cluster. In the following example, the podman commands unset these flags through a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norecover
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nobackfill
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nodown
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset pause

   Copy to Clipboard

Verification

1. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard

   Ensure the status is HEALTH_OK.

Procedure

1. Power on each Compute node.

Verification

1. Log in to each Compute as the root user.

2. Check the services on the Compute node:

   $ systemctl -t service

   Copy to Clipboard

Procedure

1. Log in as the root user to an overcloud node that runs Pacemaker.

2. Enable the STONITH device for a Compute node:

   1. Identify the Compute node STONITH device:

      # pcs stonith status

      Copy to Clipboard

   2. Clear any STONITH errors for the Compute node:

      # pcs stonith confirm <COMPUTE_NODE>

      Copy to Clipboard

      This command returns the node to a clean STONITH state.

   3. Enable the Compute node STONITH device:

      # pcs stonith enable <STONITH_DEVICE>

      Copy to Clipboard
   4. Perform these steps for each Compute node with STONITH.

3. Enable the Pacemaker remote resource on each Compute node:

   1. Identify the Pacemaker remote resources on Compute nodes:

      # pcs resource status

      Copy to Clipboard

      These resources use the ocf::pacemaker:remote agent and are usually named after the Compute node host format, such as overcloud-novacomputeiha-0.

   2. Enable each Pacemaker Remote resource. The following example shows how to enable the resource for overcloud-novacomputeiha-0:

      # pcs resource enable overcloud-novacomputeiha-0

      Copy to Clipboard
   3. Perform these steps for each Compute node with Pacemaker remote management.

4. Wait until the Pacemaker services start and check that the services started:

   # pcs status

   Copy to Clipboard

5. If any Pacemaker resources fail to start during the startup process, reset the status and the fail count of the resource:

   # pcs resource cleanup

   Copy to Clipboard
   Note

   Some services might require more time to start, such as fence_compute and fence_kdump.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the credentials file for your overcloud:

   $ source ~/overcloudrc

   Copy to Clipboard

3. View running instances in the overcloud:

   $ openstack server list --all-projects

   Copy to Clipboard

4. Start an instance in the overcloud:

   $ openstack server start <INSTANCE>

   Copy to Clipboard