Chapter 2. Preparing for an OpenStack Platform Upgrade

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

   Copy to Clipboard Toggle word wrap

3. Check the undercloud free space:

   (undercloud) $ df -h

   Copy to Clipboard Toggle word wrap

   Use the "Undercloud Reqirements" as a basis to determine if you have adequate free space.

4. If you have NTP installed on the undercloud, check that clocks are synchronized:

   (undercloud) $ sudo ntpstat

   Copy to Clipboard Toggle word wrap

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   Copy to Clipboard Toggle word wrap

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   Copy to Clipboard Toggle word wrap

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

   Copy to Clipboard Toggle word wrap

4. Check for failed containerized services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'exited=1' --all" ; done
   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'status=dead' -f 'status=restarting'" ; done

   Copy to Clipboard Toggle word wrap

5. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg'

   Copy to Clipboard Toggle word wrap

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

   Copy to Clipboard Toggle word wrap

   Replace <PASSWORD> and <IP ADDRESS> details with the respective details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

6. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec clustercheck clustercheck" ; done

   Copy to Clipboard Toggle word wrap

7. Check RabbitMQ cluster health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec $(ssh heat-admin@$NODE "sudo docker ps -f 'name=.*rabbitmq.*' -q") rabbitmqctl node_health_check" ; done

   Copy to Clipboard Toggle word wrap

8. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Copy to Clipboard Toggle word wrap

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

9. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

   Copy to Clipboard Toggle word wrap

10. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

    Copy to Clipboard Toggle word wrap

11. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

    Copy to Clipboard Toggle word wrap

12. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

    Copy to Clipboard Toggle word wrap

13. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

    Copy to Clipboard Toggle word wrap

14. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    Copy to Clipboard Toggle word wrap

    All agents should be Alive and their state should be UP.

15. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up

16. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up.