Chapter 1. About the Red Hat OpenStack Platform framework for upgrades

The following high-level changes occur during the upgrade to Red Hat OpenStack Platform (RHOSP) 17.1:

Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 17.1 to help you understand the resulting environment and any potential version-to-version changes that might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 17.1, follow these suggestions:

The Red Hat OpenStack Platform (RHOSP) 17.1 uses Red Hat Enterprise Linux (RHEL) 9.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to RHEL 9.2.

Before beginning the upgrade, review the following information to familiarize yourself with RHEL 9:

The long-life Red Hat OpenStack Platform upgrade requires a base operating system upgrade from Red Hat Enterprise Linux (RHEL) 8.4 to RHEL 9.2. The upgrade process uses the Leapp utility to perform the upgrade to RHEL 9.2. However, some aspects of the Leapp upgrade are customized to ensure that you are upgrading specifically from RHEL 8.4 to RHEL 9.2. To upgrade your operating system to RHEL 9.2, see Performing the undercloud system upgrade.

If any known limitations affect your environment, seek advice from the Red Hat Technical Support Team.

When planning to upgrade a Red Hat OpenStack Platform (RHOSP) environment from 16.2 to 17.1, consider the scope of the change. A RHOSP upgrade is similar in scope to a data center upgrade. Different firmware levels, hardware vendors, hardware profiles, networking interfaces, storage interfaces, and so on affect the upgrade process and can cause changes in behavior during the upgrade.

Review the following guidelines to adequately plan for the upgrade and increase the chance that you complete the upgrade successfully:

You can use the Red Hat OpenStack Platform (RHOSP) upgrade framework to perform an in-place upgrade path through multiple versions of the overcloud. The goal is to provide you with an opportunity to remain on certain OpenStack versions that are considered long life versions and upgrade when the next long life version is available.

The Red Hat OpenStack Platform upgrade process also upgrades the version of Red Hat Enterprise Linux (RHEL) on your nodes.

This guide provides an upgrade framework through the following versions:

For detailed support dates and information on the lifecycle support for Red Hat OpenStack Platform, see Red Hat OpenStack Platform Life Cycle.

For example, if your current deployment is Red Hat OpenStack Platform (RHOSP) 16.2.1 on Red Hat Enterprise Linux (RHEL) 8.4, you must perform a minor update to the latest RHOSP 16.2 version before you upgrade to RHOSP 17.1.

For more information, see Performing a minor update of Red Hat OpenStack Platform.

Red Hat provides two options for upgrading your environment to the next long life release:

The durations in the following table were recorded in a test environment that consisted of an overcloud with 200 nodes, and 9 Ceph Storage hosts with 17 object storage daemons (OSDs) each. The durations in the table might not apply to all production environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. Durations also depend on network I/O to container and package content, and disk I/O on the host.

To accurately estimate the upgrade duration for each task, perform these procedures in a test environment with hardware that is similar to your production environment.

Review the following known issues that might affect a successful upgrade.

If you upgrade your operating system from RHEL 7.x to RHEL 8.x, or from RHEL 8.x to RHEL 9.x, do not run a Leapp upgrade with the --debug option. The system remains in the early console in setup code state and does not reboot automatically. To avoid this issue, the UpgradeLeappDebug parameter is set to false by default. Do not change this value in your templates.

After rebooting an overcloud node, a permission issue causes collectd-sensubility to stop working. As a result, collectd-sensubility stops reporting container health. During an upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, overcloud nodes are rebooted as part of the Leapp upgrade. To ensure that collectd-sensubility continues to work, run the following command:

sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman

sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman

The Pacemaker-controlled ceph-nfs resource requires a runtime directory to store some process data. The directory is created when you install or upgrade RHOSP. Currently, a reboot of the Controller nodes removes the directory, and the ceph-nfs service does not recover when the Controller nodes are rebooted. If all Controller nodes are rebooted, the ceph-nfs service fails permanently.

You can apply the following workaround:

If the CephPools parameter is defined with a set of pool overrides, you must add rule_name: replicated_rule to the definition to avoid pool creation failures during an upgrade from RHOSP 16.2 to 17.1.

If you upgrade from Red Hat OpenStack Platform (RHOSP) 13 to 16.1 or 16.2, or from RHOSP 16.2 to 17.1, do not include the system_upgrade.yaml file in the --answers-file answer-upgrade.yaml file. If the system_upgrade.yaml file is included in that file, the environments/lifecycle/upgrade-prepare.yaml file overwrites the parameters in the system_upgrade.yaml file. To avoid this issue, append the system_upgrade.yaml file to the openstack overcloud upgrade prepare command. For example:

openstack overcloud upgrade prepare --answers-file answer-upgrade.yaml /

$ openstack overcloud upgrade prepare --answers-file answer-upgrade.yaml /
-r roles-data.yaml /
-n networking-data.yaml /
-e system_upgrade.yaml /
-e upgrade_environment.yaml /

With this workaround, the parameters that are configured in the system_upgrade.yaml file overwrite the default parameters in the environments/lifecycle/upgrade-prepare.yaml file.

During an upgrade from RHOSP 16.2 to 17.1, the operating system upgrade from RHEL 8.4 to RHEL 9.2 fails if Cinder volume NFS mounts are present on Compute nodes. Contact your Red Hat support representative for a workaround.

During an upgrade from Red Hat Ceph Storage 4 to 5, a known issue prevents Ceph Monitor nodes from being upgraded. After the first Ceph Monitor node is upgraded to version 5, the other Ceph Monitor nodes stop running and report the following message:

"FAILED ceph_assert(fs->mds_map.compat.compare(compat) == 0)"

"FAILED ceph_assert(fs->mds_map.compat.compare(compat) == 0)"

Before you upgrade your Red Hat Ceph Storage nodes, apply the workaround in the Red Hat Knowledgebase solution RHCS during upgrade RHCS 4 RHCS 5 ceph-mon is failing with "FAILED ceph_assert(fs→mds_map.compat.compare(compat) == 0)". After the upgrade is complete, the Red Hat Ceph Storage cluster is adopted by cephadm, which does not require this workaround.

In environments where the undercloud is not connected to the internet, an upgrade from Red Hat OpenStack Platform 16.2 to 17.1 fails because the infra_image value is not defined. The overcloud_upgrade_prepare.sh script tries to pull registry.access.redhat.com/ubi8/pause, which causes an error.

To avoid this issue, manually add a pause container to your Satellite server:

When you upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, the Leapp upgrade of the Red Hat Ceph Storage nodes fails because of an encrypted ceph-osd. Before you run the Leapp upgrade on your Red Hat Ceph Storage nodes, apply the workaround in the Red Hat Knowledgebase solution (FFU 16.2→17) leapp upgrade of ceph nodes is failing encrypted partition detected.

The bridge_name variable is no longer valid for nic-config templates in RHOSP 17.1. After an upgrade from RHOSP 16.2 to 17.1, if you run a stack update and the nic-config templates still include the bridge_name variable, an outage occurs. Before you upgrade to RHOSP 17.1, you need to rename the bridge_name variable.

For more information, see the Red Hat Knowledgebase solution bridge_name is still present in templates during and post FFU causing further updates failure.

If you deployed Alertmanager in a director-deployed Red Hat Ceph Storage environment, the upgrade from Red Hat Ceph Storage version 4 to version 5 fails. The failure occurs because HAProxy does not restart after you run the following command to configure cephadm on the Red Hat Ceph Storage nodes:

openstack overcloud external-upgrade run \
--skip-tags ceph_ansible_remote_tmp \
--stack <stack> \
--tags cephadm_adopt  2>&1

$ openstack overcloud external-upgrade run \
--skip-tags ceph_ansible_remote_tmp \
--stack <stack> \
--tags cephadm_adopt  2>&1

After you run the command, the Red Hat Ceph Storage cluster status is HEALTH_WARN.

For a workaround for this issue, see the Red Hat Knowledgebase solution HAProxy does not restart during RHOSP upgrade when RHCS is director-deployed and Alertmanager is enabled.

You might see a health warning message similar to the following after upgrading from Red Hat Ceph Storage 5 to 6:

[WRN] BLUESTORE_NO_PER_POOL_OMAP

[WRN] BLUESTORE_NO_PER_POOL_OMAP

You can clear this health warning message by following the instructions in the Red Hat Knowledgebase solution link: RHCS 6 - BLUESTORE_NO_PER_POOL_OMAP OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats.

If the undercloud upgrade fails, you must restart the mySQL service before you run the undercloud upgrade again. For more information about restarting the mySQL service, see the Red Hat Knowledgebase solution Update from 16.2 to 17.1 failed on migrate existing introspection data in the undercloud.

The time you will need to upgrade from Red Hat OpenStack Platform 16.2 to 17.1 increases with the number of nodes in a single role. To reduce the amount of time it takes to complete the upgrade, you can split your nodes into multiple roles. For more information, see the Red Hat Knowledgebase article How to split roles during upgrade from RHOSP 16.2 to RHOSP 17.1.

Starting with RHOSP 16.1.7, deleting the DEFAULT volume type is allowed. However, the DEFAULT volume type is hard coded in the cinder.conf file, and therefore it must exist during a fast forward upgrade. If you deleted the DEFAULT volume type, do not perform an upgrade from RHOSP 16.2 to RHOSP 17.1 until after you perform the workaround described in the Red Hat Knowledgebase solution Performing online database updates failed.

When you upgrade from RHOSP 16.2 to 17.1, during the system upgrade, a known issue causes GRUB to contain RHEL 7 entries instead of RHEL 8 entries. As a result, the hosts cannot reboot. This issue affects environments that previously ran RHOSP 13.0 or earlier.

Workaround: See the Red Hat Knowledgebase solution Openstack 16 to 17 FFU - During LEAPP upgrade UEFI systems do not boot due to invalid /boot/grub2/grub.cfg.

The Leapp version that upgrades Red Hat Enterprise Linux 8.4 to 9.2 does not verify whether all partitions have enough disk space. Before you perform the Red Hat OpenStack Platform system upgrade, you must manually check that all partitions have at least 3 GB of disk space. Failure to do so can cause the node to reboot and enter into an emergency shell.

If you perform an upgrade of your RHOSP environment to 17.1.x, the pre-upgrade package_version validation fails because the validation cannot find a matching podman version.

Workaround: To skip the package_version validation, use the --skiplist package-version option when you run the pre-upgrade validation:

validation run -i inventory.yaml --group pre-upgrade --skiplist package-version

$ validation run -i inventory.yaml --group pre-upgrade --skiplist package-version

Static file compression does not run automatically after an upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1. As a result, the missing static files cause the Red Hat OpenStack Platform (RHOSP) dashboard (horizon) to fail. To run the compression manually after the upgrade, see Compressing Red Hat OpenStack Platform dashboard files.

If you are using director-deployed Red Hat Ceph Storage 5 nodes, during an upgrade from RHOSP 16.2 to 17.1, the EUS repositories that are specified in the UpgradeInitCommand parameter override the repositories in the Red Hat Ceph Storage role.

Workaround: To use the repositories listed in your Red Hat Ceph Storage nodes, add the following parameters:

In RHOSP 17.1, you can use the net_config_override variable in the undercloud.conf file to identify an alternate network configuration file. In that alternate file, you must include the IP addresses that are used for the Virtual IPs (VIPs). If the IP addresses are not present in that file, when you run openstack undercloud install, the DB sync hangs. For example:

network_config:
       name: br-ctlplane
       type: ovs_bridge
       use_dhcp: false
       dns_servers:
       172.16.0.1
       10.0.0.1
       domain: lab.example.com
       ovs_extra:
       "br-set-external-id br-ctlplane bridge-id br-ctlplane"
       addresses:
       ip_netmask: 192.168.24.1/24
       ip_netmask: 192.168.24.2/32 <---------------------------|
       ip_netmask: 192.168.24.3/32 <---------------------------|
       members:
       type: interface
       name: eth0

network_config:
       name: br-ctlplane
       type: ovs_bridge
       use_dhcp: false
       dns_servers:
       172.16.0.1
       10.0.0.1
       domain: lab.example.com
       ovs_extra:
       "br-set-external-id br-ctlplane bridge-id br-ctlplane"
       addresses:
       ip_netmask: 192.168.24.1/24
       ip_netmask: 192.168.24.2/32 <---------------------------|
       ip_netmask: 192.168.24.3/32 <---------------------------|
       members:
       type: interface
       name: eth0

An upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1 fails if additional, non-core openvswitch packages are installed.

Workaround: See the Red Hat Knowledgebase solution FFU is failing on Ansible task special treatment for OpenvSwitch.

After completing an upgrade from 16.2 to 17.1 and Red Hat Ceph Storage 7, no data (N/A) is displayed in the Overall performance section of the OSD dashboard.

Workaround: See the Red Hat Knowledgebase solution Ceph dashboard is not showing performance metrics graphs on RHCS 6.

After an upgrade from Red Hat Ceph Storage 6 to 7, if you have a disconnected Red Hat OpenStack environment, Grafana attempts to access the internet to download updates. As a result, Grafana times out.

Workaround: See BZ#2346107.

If you attempt to perform a Leapp OS upgrade with NVIDIA drivers, the system upgrade fails with the following error in /var/log/leapp/leapp-report.txt:

Summary: Leapp has detected that the NVIDIA proprietary driver has been loaded, which also means the nouveau driver is blacklisted. If you upgrade now, you will end up without a graphical session, as the newer kernel won't be able to load the NVIDIA driver module and nouveau will still be blacklisted.

Please uninstall the NVIDIA graphics driver before upgrading to make sure you have a graphical session after upgrading.

Summary: Leapp has detected that the NVIDIA proprietary driver has been loaded, which also means the nouveau driver is blacklisted. If you upgrade now, you will end up without a graphical session, as the newer kernel won't be able to load the NVIDIA driver module and nouveau will still be blacklisted.

Please uninstall the NVIDIA graphics driver before upgrading to make sure you have a graphical session after upgrading.

Workaround:

During an upgrade from RHOSP 16.2 to 17.1, the overcloud upgrade fails because the environment uses the following unsupported service in the upgrade_tasks_step3.yaml file: OS::TripleO::Services::OVNDBs: deployment/ovn/ovn-dbs-pacemaker-puppet.yaml

Workaround: