Upgrading Red Hat OpenStack Platform

Before performing the update, Red Hat advises the following:

The director relies on standard RPM methods to update your environment. This involves ensuring your director’s host uses the latest packages through yum.

Perform a reboot of the node to enable new system settings and refresh all undercloud services:

When the node boots, check the status of all services:

$ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

Verify the existence of your Overcloud and its nodes:

$ source ~/stackrc
$ openstack server list
$ openstack baremetal node list
$ openstack stack list

It is important to keep your overcloud images up to date to ensure the image configuration matches the requirements of the latest openstack-tripleo-heat-template package. To ensure successful deployments and scaling operations in the future, update your overclouds images using the instructions in Section 2.2.2, “Updating the Overcloud Images”.

The Undercloud update process might download new image archives from the rhosp-director-images and rhosp-director-images-ipa packages. Check the yum log to determine if new image archives are available:

$ sudo grep "rhosp-director-images" /var/log/yum.log

If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

$ rm -rf ~/images/*

Extract the archives:

$ cd ~/images
$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-11.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-11.0.tar; do tar -xvf $i; done

Import the latest images into the director and configure nodes to use the new images

$ cd ~
$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
$ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

To finalize the image update, verify the existence of the new images:

$ openstack image list
$ ls -l /httpboot

The director is now updated and using the latest images. You do not need to restart any services after the update.

The Overcloud relies on standard RPM methods to update the environment. This involves two steps:

This starts update process. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example:

not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-compute-0']
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode:

Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a Python-based regular expression to clear breakpoints on multiple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have completed their update.

The update command reports a COMPLETE status when the update completes:

...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
COMPLETE
update finished with status COMPLETE

If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes:

$ sudo pcs property set stonith-enabled=true

The update process does not reboot any nodes in the Overcloud automatically. Major and minor version updates to the kernel or Open vSwitch require a reboot, such as when your overcloud operating system updates from Red Hat Enterprise Linux 7.2 to 7.3, or Open vSwitch from version 2.4 to 2.5. Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If they have, reboot each node using the "Rebooting the Overcloud' procedures in the Director Installation and Usage guide.

To upgrade the Red Hat OpenStack Platform director, follow this procedure:

Perform a reboot of the node to enable new system settings and refresh all undercloud services:

When the node boots, check the status of all services:

$ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

Verify the existence of your Overcloud and its nodes:

$ source ~/stackrc
$ openstack server list
$ openstack baremetal node list
$ openstack stack list

If necessary, review the configuration files on the director. The upgraded packages might have installed .rpmnew files appropriate to the Red Hat OpenStack Platform 11 version of the service.

This procedure ensures you have the latest images for node discovery and Overcloud deployment. The new images from the rhosp-director-images and rhosp-director-images-ipa packages are already updated from the Undercloud upgrade.

Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

$ rm -rf ~/images/*

Extract the archives:

$ cd ~/images
$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-11.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-11.0.tar; do tar -xvf $i; done

Import the latest images into the director and configure nodes to use the new images

$ cd ~
$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
$ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

To finalize the image update, verify the existence of the new images:

$ openstack image list
$ ls -l /httpboot

The director is now upgraded with the latest images.

The upgrade process installs a new set of core Heat templates that correspond to the latest overcloud version. Red Hat OpenStack Platform’s repository retains the previous version of the core template collection in the openstack-tripleo-heat-templates-compat package. You install this package with the following command:

$ sudo yum install openstack-tripleo-heat-templates-compat

This installs the previous templates in the compat directory of your Heat template collection (/usr/share/openstack-tripleo-heat-templates/compat) and also creates a link to compat named after the previous version (newton). These templates are backwards compatible with the upgraded director, which means you can use the latest version of the director to install an overcloud of the previous version.

Comparing the previous version with the latest version helps identify changes to the overcloud during the upgrade. If you need to compare the current template collection with the previous version, use the following process:

This shows the core template changes from one version to the next. These changes provide an idea of what should occur during the overcloud upgrade.

Before upgrading the overcloud, update its subscription details to ensure your environment uses the latest repositories. If using an environment file for Satellite registration, update the following parameters in the environment file:

For more information and examples of the environment file format, see "Overcloud Registration" in the Advanced Overcloud Customization guide.

The following sections apply if using a custom roles_data.yaml file to define your overcloud roles.

Remove the following deprecated services from your custom roles_data.yaml file:

Add the following new services to your custom roles_data.yaml file:

Update any additional parts of the overcloud that might require these new services such as:

The upgrade process provides staged upgrades for each composable service on all roles. However, some roles require an individual upgrade of their nodes to ensure availability of instances and service. These roles are:

This upgrade process uses the upgrade-non-controller.sh command to upgrade nodes in these roles.

The default roles_data.yaml file for Red Hat OpenStack Platform 11 marks these roles with the disable_upgrade_deployment: True, which excludes these roles during the main composable service upgrade process. This provides a method for upgrading the nodes in these roles individually. However, if using a custom roles_data.yaml file that contains these roles, make sure the Compute and ObjectStorage role definitions contain the disable_upgrade_deployment: True parameter. For example:

- name: Compute
  CountDefault: 1
  disable_upgrade_deployment: True
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
...

Use the disable_upgrade_deployment for other custom roles that require a manual upgrade, such as custom Compute roles.

Some storage backends have changed from using configuration hooks to their own composable service. If using a custom storage backend, check the associated environment file in the environments directory for new parameters and resources. Update any custom environment files for your backends. For example:

For example, the NetApp Block Storage (cinder) backend used the following resources for these respective versions:

As a result, you now use the new OS::TripleO::Services::CinderBackendNetApp resource and its associated service template for this backend.

Follow these guidelines to upgrade from Red Hat OpenStack Platform 10 to Red Hat OpenStack Platform 11 when you have OVS-DPDK configured.

Begin your upgrade preparations based on the guidance from Upgrading Red Hat OpenStack Platform. Before you deploy the overcloud based on Upgrading Composable Services, follow these steps:

Note the following information about overcloud parameters for upgrades:

If using a modified version of the core Heat template collection from Red Hat OpenStack Platform 10, you need to re-apply your customizations to a copy of the Red Hat OpenStack Platform 11 version. To do this, use a git version control system similar to the one outlined in "Using Customized Core Heat Templates" from the Advanced Overcloud Customization guide.

Red Hat provides updates to the Heat template collection over subsequent releases. Using a modified template collection without a version control system can lead to a divergence between your custom copy and the original copy in /usr/share/openstack-tripleo-heat-templates.

As an alternative to using a custom Heat template collection, Red Hat recommends using the Configuration Hooks from the Advanced Overcloud Customization guide.

An overcloud upgrade requires an additional environment file (major-upgrade-composable-steps.yaml) to your deployment. This file provides a full upgrade to all nodes except roles marked with the disable_upgrade_deployment: True parameter.

Run the openstack overcloud deploy command from your undercloud and include the major-upgrade-composable-steps.yaml environment file. Include all options and custom environment files relevant to your environment, such as network isolation and storage.

The following is an example of an openstack overcloud deploy command with both the required and optional files:

$ openstack overcloud deploy --templates \
  --control-scale 3 \
  --compute-scale 3 \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
  -e network_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml \
  --ntp-server pool.ntp.org

Wait until the Overcloud updates with the new environment file’s configuration.

Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If so, perform a reboot of each node, use the reboot instructions from "Rebooting the Overcloud" in the Director Installation and Usage guide

During the deployment of the major-upgrade-composable-steps.yaml environment file, the director passes a special upgrade script to each node in roles marked with the disable_upgrade_deployment: True parameter. The next few sections show how to invoke this script from the undercloud and upgrade the remaining roles.

The director uses the upgrade-non-controller.sh command to run the upgrade script passed to the Object Storage nodes from the major-upgrade-composable-steps.yaml environment file. For this step, upgrade each Object Storage node using the following command:

$ for NODE in `openstack server list -c Name -f value --name objectstorage` ; do upgrade-non-controller.sh --upgrade $NODE ; done

Upgrading each Object Storage node individually ensures the service remains available during the upgrade.

Wait until each Object Storage node completes its upgrade.

Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If so, perform a reboot of each node:

Upgrade each Compute node individually and ensure zero downtime of instances in your OpenStack Platform environment. This involves the following workflow:

List all Compute nodes and their UUIDs:

$ source ~/stackrc
$ openstack server list | grep "compute"

Select a Compute node to upgrade and first migrate its instances using the following process:

The director uses the upgrade-non-controller.sh command to run the upgrade script passed to each non-Controller node from the major-upgrade-composable-steps.yaml environment file. Upgrade each Compute node with the following command:

$ source ~/stackrc
$ upgrade-non-controller.sh --upgrade [NODE]

Replace [NODE] with the UUID or name of the chosen Compute node. Wait until the Compute node completes its upgrade.

Check the /var/log/yum.log file on the Compute node you have upgraded to see if one of the following packages have updated their major or minor versions:

If so, perform a reboot of the node:

Repeat this process for each node individually until you have upgraded and rebooted all nodes.

After upgrading all Compute nodes, revert back to the stackrc access details:

$ source ~/stackrc

The director needs to run through the upgrade finalization to ensure the Overcloud stack is synchronized with the current Heat template collection. This involves an environment file (major-upgrade-converge.yaml), which you include using the openstack overcloud deploy command.

parameter_defaults:
   RbdDefaultFeatures: 1

Run the openstack overcloud deploy from your Undercloud and include the major-upgrade-converge.yaml environment file. Make sure you also include all options and custom environment files relevant to your environment, such as backwards compatibility for Ceph (if applicable), network isolation, and storage.

This following is an example of an openstack overcloud deploy command with the added major-upgrade-converge.yaml file:

$ openstack overcloud deploy --templates \
  --control-scale 3 \
  --compute-scale 3 \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
  -e network_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-converge.yaml \
  --ntp-server pool.ntp.org

Wait until the Overcloud updates with the new environment file’s configuration.