Upgrading Red Hat OpenStack Platform

Procedure

1. Log into the undercloud as the root user.

2. Back up the database:

   [root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql

   Copy to Clipboard

3. Create a backup directory and change the user ownership of the directory to the stack user:

   [root@director ~]# mkdir /backup
   [root@director ~]# chown stack: /backup

   Copy to Clipboard

   You will use this directory to store the archive containing the undercloud database and file system.

4. Change to the backup directory

   [root@director ~]# cd /backup

   Copy to Clipboard

5. Archive the database backup and the configuration files:

   [root@director ~]# tar --xattrs --ignore-failed-read -cf \
       undercloud-backup-`date +%F`.tar \
       /etc \
       /var/log \
       /var/lib/glance \
       /var/lib/certmonger \
       /srv/node \
       /root \
       /home/stack

   Copy to Clipboard
   • The --ignore-failed-read option skips any directory that does not apply to your undercloud.
   • The --xattrs option includes extended attributes, which are required to store metadata for Object Storage (swift).

   This creates a file named undercloud-backup-<date>.tar.gz, where <date> is the system date. Copy this tar file to a secure location.

Procedure

1. Perform the database backup:

   1. Log into a Controller node. You can access the overcloud from the undercloud:

      $ ssh heat-admin@192.0.2.100

      Copy to Clipboard

   2. Change to the root user:

      $ sudo -i

      Copy to Clipboard

   3. Install the mariadb client tools if they are not installed already:

      # dnf install mariadb

      Copy to Clipboard

   4. Create a temporary directory to store the backups:

      # mkdir -p /var/tmp/mysql_backup/

      Copy to Clipboard

   5. Obtain the database password and store it in the MYSQLDBPASS environment variable. The password is stored in the mysql::server::root_password variable within the /etc/puppet/hieradata/service_configs.json file. Use the following command to store the password:

      # MYSQLDBPASS=$(sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

      Copy to Clipboard

   6. Backup the database:

      # mysql -uroot -p$MYSQLDBPASS -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot -p$MYSQLDBPASS --single-transaction --databases > /var/tmp/mysql_backup/openstack_databases-`date +%F`-`date +%T`.sql

      Copy to Clipboard

      This dumps a database backup called /var/tmp/mysql_backup/openstack_databases-<date>.sql where <date> is the system date and time. Copy this database dump to a secure location.

   7. Backup all the users and permissions information:

      # mysql -uroot -p$MYSQLDBPASS -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -uroot -p$MYSQLDBPASS -s -N -e | sed 's/$/;/' > /var/tmp/mysql_backup/openstack_databases_grants-`date +%F`-`date +%T`.sql

      Copy to Clipboard

      This dumps a database backup called /var/tmp/mysql_backup/openstack_databases_grants-<date>.sql where <date> is the system date and time. Copy this database dump to a secure location.

2. Backup the Pacemaker configuration:

   1. Log into a Controller node.

   2. Run the following command to create an archive of the current Pacemaker configuration:

      # sudo pcs config backup pacemaker_controller_backup

      Copy to Clipboard
   3. Copy the resulting archive (pacemaker_controller_backup.tar.bz2) to a secure location.

3. Backup the Redis cluster:

   1. Obtain the Redis endpoint from HAProxy:

      # REDISIP=$(sudo hiera -c /etc/puppet/hiera.yaml redis_vip)

      Copy to Clipboard

   2. Obtain the master password for the Redis cluster:

      # REDISPASS=$(sudo hiera -c /etc/puppet/hiera.yaml redis::masterauth)

      Copy to Clipboard

   3. Check connectivity to the Redis cluster:

      # redis-cli -a $REDISPASS -h $REDISIP ping

      Copy to Clipboard

   4. Dump the Redis database:

      # redis-cli -a $REDISPASS -h $REDISIP bgsave

      Copy to Clipboard

      This stores the database backup in the default /var/lib/redis/ directory. Copy this database dump to a secure location.

4. Backup the filesystem on each Controller node:

   1. Create a directory for the backup:

      # mkdir -p /var/tmp/filesystem_backup/

      Copy to Clipboard

   2. Run the following tar command:

      # tar --ignore-failed-read --xattrs \
          -zcvf /var/tmp/filesystem_backup/fs_backup-`date '+%Y-%m-%d-%H-%M-%S'`.tar.gz \
          /var/lib/kolla \
          /var/lib/config-data \
          /var/log/containers \
          /var/log/openvswitch \
          /etc \
          /srv/node \
          /root \
          /home/heat-admin

      Copy to Clipboard

      The --ignore-failed-read option ignores any missing directories, which is useful if certain services are not used or are separated on their own custom roles.

5. Copy the resulting tar file to a secure location.

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'tripleo_*'

   Copy to Clipboard

3. Check the undercloud free space:

   (undercloud) $ df -h

   Copy to Clipboard

   Use the "Undercloud Reqirements" as a basis to determine if you have adequate free space.

4. If you have NTP installed on the undercloud, check that clocks are synchronized:

   (undercloud) $ sudo chronyc -a 'burst 4/4'

   Copy to Clipboard

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   Copy to Clipboard

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   Copy to Clipboard

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   Copy to Clipboard

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'tripleo_*' 'ceph*'" ; done

   Copy to Clipboard

4. Check for failed containerized services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo podman ps -f 'exited=1' --all" ; done

   Copy to Clipboard

5. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg'

   Copy to Clipboard

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | cut -d, -f 1,2,18,37,57 | column -s, -t

   Copy to Clipboard

   Replace <PASSWORD> and <IP ADDRESS> details with the actual details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

   Note

   In case the nodes run Redis services, only one node displays an ON status for that service. This is because Redis is an active-passive service, which runs only on one node at a time.

6. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo podman exec clustercheck clustercheck" ; done

   Copy to Clipboard

7. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Copy to Clipboard

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

8. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

   Copy to Clipboard

9. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

   Copy to Clipboard

10. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

    Copy to Clipboard

11. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo chronyc -a 'burst 4/4'" ; done

    Copy to Clipboard

12. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

    Copy to Clipboard

13. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    Copy to Clipboard

    All agents should be Alive and their state should be UP.

14. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    Copy to Clipboard

    All agents' status should be enabled and their state should be up

15. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    Copy to Clipboard

    All agents' status should be enabled and their state should be up.

Procedure

1. Check the current list of hardware types enabled:

   $ source ~/stackrc
   $ openstack baremetal driver list --type dynamic

   Copy to Clipboard

2. If you use a hardware type driver that is not enabled, enable the driver using the enabled_hardware_types parameter in the undercloud.conf file:

   enabled_hardware_types = ipmi,redfish,idrac

   Copy to Clipboard

3. Save the file and refresh the undercloud:

   $ openstack undercloud install

   Copy to Clipboard

4. Run the following commands, substituting the OLDDRIVER and NEWDRIVER variables for your power management type:

   $ source ~/stackrc
   $ OLDDRIVER="pxe_ipmitool"
   $ NEWDRIVER="ipmi"
   $ for NODE in $(openstack baremetal node list --driver $OLDDRIVER -c UUID -f value) ; do openstack baremetal node set $NODE --driver $NEWDRIVER; done

   Copy to Clipboard

Procedure

1. Log in to the director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-13-rpms

   Copy to Clipboard

3. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-14-rpms

   Copy to Clipboard

4. Run dnf to upgrade the director’s main packages:

   $ sudo dnf update -y python-tripleoclient* openstack-tripleo-common openstack-tripleo-heat-templates

   Copy to Clipboard

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   Copy to Clipboard

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means the director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. The director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can also use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Edit the containers-prepare-parameter.yaml and make the modifications to suit your requirements.

Procedure

1. Run the following command to upgrade the director on the undercloud:

   $ openstack undercloud upgrade

   Copy to Clipboard

   This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

   Note

   The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the -y option:

   $ openstack undercloud upgrade -y

   Copy to Clipboard

2. The script also starts all OpenStack Platform service containers on the undercloud automatically. Check the enabled containers using the following command:

   [stack@director ~]$ sudo docker ps

   Copy to Clipboard

3. The script adds the stack user to the docker group to ensure that the stack user has access to container management commands. Refresh the stack user permissions with the following command:

   [stack@director ~]$ exec su -l stack

   Copy to Clipboard

   The command prompts you to log in again. Enter the stack user password.

4. To initialize the stack user to use the command line tools, run the following command:

   [stack@director ~]$ source ~/stackrc

   Copy to Clipboard

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

   (undercloud) [stack@director ~]$

   Copy to Clipboard

Procedure

1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

   Copy to Clipboard

2. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-14.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-14.0.tar; do tar -xvf $i; done
   $ cd ~

   Copy to Clipboard

3. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

   Copy to Clipboard

4. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

   Copy to Clipboard

5. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /var/lib/ironic/httpboot/

   Copy to Clipboard

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

   Copy to Clipboard
3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Change to the director’s core template collection:

   $ cd /usr/share/openstack-tripleo-heat-templates

   Copy to Clipboard

2. Run the merge-new-params-nic-config-script.py python script in the tools directory. For example, to update a custom Controller node NIC template, run the script with the following options:

   $ python ./tools/merge-new-params-nic-config-script.py --role-name Controller -t /home/stack/ccsosp-templates/custom-nics/controller.yaml

   Copy to Clipboard

   Note the following options used with this script:

   • --role-name defines the name of the role to use as a basis for the template update.
   • -t, --template defines the filename of the NIC template to update.
   • -n, --network-data defines the relative path to the network_data file. Use this option for custom network_data files. If omitted, the script uses the default file.
   • -r, --roles-data, defines the relative path to the roles_data.yaml file. Use this option for custom roles_data files. If omitted, the script uses the default file.

3. The script saves a copy of the original template and adds a timestamp extension to the copy’s filename. To compare the differences between the original and updated template, run the following command:

   $ diff /home/stack/ccsosp-templates/custom-nics/controller.yaml.[TIMESTAMP] /home/stack/ccsosp-templates/custom-nics/controller.yaml

   Copy to Clipboard

   Replace [TIMESTAMP] with the timestamp on the original filename.

   The output displays the new route parameters for that role:

   StorageMgmtInterfaceRoutes:
     default: []
     description: >
       Routes for the storage_mgmt network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json
   TenantInterfaceRoutes:
     default: []
     description: >
       Routes for the tenant network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json
   ExternalInterfaceRoutes:
     default: []
     description: >
       Routes for the external network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json
   InternalApiInterfaceRoutes:
     default: []
     description: >
       Routes for the internal_api network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json
   ManagementInterfaceRoutes:
     default: []
     description: >
       Routes for the management network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json
   StorageInterfaceRoutes:
     default: []
     description: >
       Routes for the storage network traffic. JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}] Unless
       the default is changed, the parameter is automatically resolved from the subnet host_routes attribute.
     type: json

   Copy to Clipboard

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --templates \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e)
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the openstack overcloud external-upgrade run command against tasks tagged with the container_image_prepare tag:

   $ openstack overcloud external-upgrade run --tags container_image_prepare

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. If you use monolithic Controller nodes, run the upgrade command against the Controller role:

   $ openstack overcloud upgrade run --nodes Controller

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.

3. If you use Controller services split across multiple roles:

   1. Run the upgrade command for roles with Pacemaker services:

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the upgrade command:

   $ openstack overcloud upgrade run --nodes Compute

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the Compute node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the upgrade command:

   $ openstack overcloud upgrade run --nodes CephStorage

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the node upgrade completes.

4. Run the Ceph Storage external upgrade process. For example:

   $ openstack overcloud external-upgrade run --tags ceph

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.
   • To pass any additional overrides, see in Section 5.6.1, “Custom parameters for upgrades”.
5. Wait until the Ceph Storage node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the openstack overcloud external-upgrade run command against tasks tagged with the online_upgrade tag:

   $ openstack overcloud external-upgrade run --tags online_upgrade

   Copy to Clipboard
   • If you use a custom stack name, pass the name with the --stack option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard

2. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --templates \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e).
   • If you use a custom stack name, pass the name with the --stack option.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
3. Wait until the upgrade finalization completes.