Chapter 8. Executing Post Upgrade Steps

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

   Copy to Clipboard Toggle word wrap

3. Check the undercloud free space:

   (undercloud) $ df -h

   Copy to Clipboard Toggle word wrap

   Use the "Undercloud Requirements" as a basis to determine if you have adequate free space.

4. If you have NTP installed on the undercloud, check that clocks are synchronized:

   (undercloud) $ sudo ntpstat

   Copy to Clipboard Toggle word wrap

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   Copy to Clipboard Toggle word wrap

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   Copy to Clipboard Toggle word wrap

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

   Copy to Clipboard Toggle word wrap

4. Check for failed containerized services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'exited=1' --all" ; done
   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'status=dead' -f 'status=restarting'" ; done

   Copy to Clipboard Toggle word wrap

5. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg'

   Copy to Clipboard Toggle word wrap

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | cut -d, -f 1,2,18,37,57 | column -s, -t

   Copy to Clipboard Toggle word wrap

   Replace <PASSWORD> and <IP ADDRESS> details with the actual details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

   Note

   In case the nodes run Redis services, only one node displays an ON status for that service. This is because Redis is an active-passive service, which runs only on one node at a time.

6. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec clustercheck clustercheck" ; done

   Copy to Clipboard Toggle word wrap

7. Check RabbitMQ cluster health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec $(ssh heat-admin@$NODE "sudo docker ps -f 'name=.*rabbitmq.*' -q") rabbitmqctl node_health_check" ; done

   Copy to Clipboard Toggle word wrap

8. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Copy to Clipboard Toggle word wrap

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

9. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

   Copy to Clipboard Toggle word wrap

10. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

    Copy to Clipboard Toggle word wrap

11. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

    Copy to Clipboard Toggle word wrap

12. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

    Copy to Clipboard Toggle word wrap

13. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

    Copy to Clipboard Toggle word wrap

14. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    Copy to Clipboard Toggle word wrap

    All agents should be Alive and their state should be UP.

15. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up

16. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up.

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

   Copy to Clipboard Toggle word wrap

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-13.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-13.0.tar; do tar -xvf $i; done
   $ cd ~

   Copy to Clipboard Toggle word wrap

4. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

   Copy to Clipboard Toggle word wrap

5. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

   Copy to Clipboard Toggle word wrap

6. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the deploy command and include all environment files relevant to your overcloud:

   $ openstack overcloud deploy \
       --templates \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files using -e.
   • If applicable, your custom roles (roles_data) file using --roles-file.
3. Wait until the deployment completes.