Framework for Upgrades (13 to 16.1)

1. Before you can upgrade your Red Hat OpenStack Platform deployment from version 13 to version 16.1, you must upgrade your Red Hat Ceph Storage cluster from version 3 to version 4. For more information, see Upgrading a Red Hat Ceph Storage cluster in the Red Hat Ceph Storage 4 Installation Guide.
2. After you upgrade your Red Hat Ceph Storage cluster from version 3 to version 4, Red Hat OpenStack Platform 13 might still run RHCSv3 client components, however these are compatible against the RHCSv4 cluster.
3. You can follow the upgrade path described in the Framework For Upgrades (13 to 16.1) document, and where applicable, you must complete the conditional steps that support this particular scenario. A conditional step starts with the following statement: "If you are upgrading with external Ceph deployments".
4. When you upgrade with external Ceph deployments, you install RHCSv4 ceph-ansible as part of the overcloud upgrade process. When you upgrade a Ceph cluster that was deployed using director, you install RHCSv4 ceph-ansible after the overcloud upgrade process is complete.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a bash script called pre-upgrade-validations.sh and include the following content in the script:

   #!/bin/bash
   for VALIDATION in $(openstack action execution run tripleo.validations.list_validations '{"groups": ["pre-upgrade"]}' | jq ".result[] | .id")
   do
     echo "=== Running validation: $VALIDATION ==="
     STACK_NAME=$(openstack stack list -f value -c 'Stack Name')
     ID=$(openstack workflow execution create -f value -c ID tripleo.validations.v1.run_validation "{\"validation_name\": $VALIDATION, \"plan\": \"$STACK_NAME\"}")
     while [ $(openstack workflow execution show $ID -f value -c State) == "RUNNING" ]
     do
       sleep 1
     done
     echo ""
     openstack workflow execution output show $ID | jq -r ".stdout"
     echo ""
   done

4. Add permission to run the script:

   $ chmod +x pre-upgrade-validations.sh

5. Run the script:

   $ ./pre-upgrade-validations.sh

   Review the script output to determine which validations succeed and fail:

   === Running validation: "check-ftype" ===
   
   Success! The validation passed for all hosts:
   * undercloud

Procedure

1. Log in to the undercloud as the stack user.

2. Create an Ansible playbook named playbook-nics.yaml and copy the following content into the playbook:

   ---
   - name: Rename eth devices
     hosts: all
     become: yes
     vars:
       prefix: "em"
       undercloud_conf: "/home/stack/undercloud.conf"
       osnet_conf: "/etc/os-net-config/config.json"
     tasks:
       - set_fact:
           eth_interfaces: "{{ ansible_interfaces | select('match','eth.*') | list }}"
       - debug:
           msg: "{{ eth_interfaces }}"
       - name: Update udev rules
         lineinfile:
           line: "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"{{ ansible_facts[item]['perm_macaddress'] | default(ansible_facts[item]['macaddress']) }}\", NAME=\"{{ item|replace('eth',prefix) }}\""
           path: /etc/udev/rules.d/70-rhosp-persistent-net.rules
           create: True
         with_items: "{{ eth_interfaces }}"
       - name: Rename eth files
         block:
           - name: Check that eth files exists
             stat:
               path: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
             register: nic_result
             with_items: "{{ eth_interfaces }}"
           - name: Copy nic files using the new prefix
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Edit NAME in new network-script files
             lineinfile:
               regexp: "^NAME=.*"
               line: "NAME={{ item.item|replace('eth',prefix) }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Edit DEVICE in new network-script files
             lineinfile:
               regexp: "^DEVICE=.*"
               line: "DEVICE={{ item.item|replace('eth',prefix) }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Backup old eth network-script files
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path }}.bak"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Remove old eth network-script files
             file:
               path: "{{ item.stat.path }}"
               state: absent
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
       - name: Rename route files
         block:
           - name: Check that route files exists
             stat:
               path: /etc/sysconfig/network-scripts/route-{{ item }}
             register: route_result
             with_items: "{{ eth_interfaces }}"
           - name: Copy route files using the new prefix
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Update prefix in route files that use IP command arguments format
             replace:
               regexp: "eth"
               replace: "{{ prefix }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Backup old route files
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path }}.bak"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Remove old route files
             file:
               path: "{{ item.stat.path }}"
               state: absent
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
       - name: Perform a final regex for any remaining eth prefixes in ifcfg files
         block:
           - name: Get a list of all ifcfg files
             find:
               paths: /etc/sysconfig/network-scripts/
               patterns: 'ifcfg-*'
               excludes: '*.bak'
             register: ifcfg_files
           - name: Perform final regex on ifcfg files
             replace:
               path: "{{ item[0].path }}"
               regexp: "{{ item[1] }}"
               replace: "{{ item[1]|replace('eth',prefix) }}"
             with_nested:
               - "{{ ifcfg_files.files }}"
               - "{{ eth_interfaces }}"
       - name: Replace interface name in files referencing old eth interface
         block:
           - name: Check if undercloud.conf exists
             stat:
               path: "{{ undercloud_conf }}"
             register: undercloud_conf_stat
           - name: Replace interface name in undercloud.conf
             replace:
               path: "{{ undercloud_conf }}"
               regexp: 'eth(\d+)'
               replace: "{{ prefix }}\\1"
             when: undercloud_conf_stat.stat.exists
           - name: Check if os-net-config's config.json exists
             stat:
               path: "{{ osnet_conf }}"
             register: osnet_conf_stat
           - name: Replace interface name in config.json
             replace:
               path: "{{ osnet_conf }}"
               regexp: 'eth(\d+)'
               replace: "{{ prefix }}\\1"
             when: osnet_conf_stat.stat.exists
       - name: Patch vlan devices
         block:
           - name: Check that vlan files exists
             stat:
               path: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
             register: nic_result
             when: item.startswith("vlan")
             with_items: "{{ ansible_interfaces }}"
           - name: Backup old vlan network-script files
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path }}.bak"
             when: item.item.startswith("vlan") and item.stat.exists
             with_items: "{{ nic_result.results }}"
           - name: Edit PHYSDEV in new network-script files
             replace:
               path: "{{ item.stat.path }}"
               regexp: "^PHYSDEV=eth"
               replace: "PHYSDEV={{ prefix }}"
             when: item.item.startswith("vlan") and item.stat.exists
             with_items: "{{ nic_result.results }}"

   Note

   You will use this playbook to rename the overcloud NICs at a later stage in the upgrade process.

3. Run the playbook-nics.yaml playbook on the undercloud:

   $ ansible-playbook -c local -i localhost, playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -c local -i localhost, -e prefix="mynic" ~/playbook-nics.yaml

   The NIC changes are only applied after the Leapp upgrade process completes and the node is rebooted.

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

   $ sudo grep PermitRootLogin /etc/ssh/sshd_config

3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

   PermitRootLogin no

4. Save the file.

Procedure

1. Check the current list of hardware types enabled:

   $ source ~/stackrc
   $ openstack baremetal driver list --type dynamic

2. If you use a hardware type driver that is not enabled, enable the driver using the enabled_hardware_types parameter in the undercloud.conf file:

   enabled_hardware_types = ipmi,redfish,idrac

3. Save the file and refresh the undercloud:

   $ openstack undercloud install

4. Run the following commands, substituting the OLDDRIVER and NEWDRIVER variables for your power management type:

   $ source ~/stackrc
   $ OLDDRIVER="pxe_ipmitool"
   $ NEWDRIVER="ipmi"
   $ for NODE in $(openstack baremetal node list --driver $OLDDRIVER -c UUID -f value) ; do openstack baremetal node set $NODE --driver $NEWDRIVER; done

Procedure

1. Log in to the undercloud as the stack user.

2. Disable the main OpenStack services on the undercloud:

   $ sudo systemctl stop 'openstack-*' httpd haproxy mariadb 'rabbitmq*' docker xinetd

3. Remove the main OpenStack services from the undercloud, except OpenvSwitch and certain Python 2 packages that are required for the upgrade:

   $ sudo yum -y remove '*el7ost*' 'galera*' 'haproxy*' \
       httpd 'mysql*' 'pacemaker*' xinetd python-jsonpointer \
       qemu-kvm-common-rhev qemu-img-rhev 'rabbit*' \
       'redis*' \
       -- \
       -'*openvswitch*' -python-docker -python-PyMySQL \
       -python-pysocks -python2-asn1crypto -python2-babel \
       -python2-cffi -python2-cryptography -python2-dateutil \
       -python2-idna -python2-ipaddress -python2-jinja2 \
       -python2-jsonpatch -python2-markupsafe -python2-pyOpenSSL \
       -python2-requests -python2-six -python2-urllib3 \
       -python-httplib2 -python-passlib -python2-netaddr -ceph-ansible

4. Remove the content from the /etc/httpd and /var/lib/docker directories:

   $ sudo rm -rf /etc/httpd /var/lib/docker

Procedure

1. Log in to the undercloud as the stack user.

2. Install the Leapp utility and jq:

   $ sudo yum install leapp
   $ sudo yum install jq

3. Download the additional required data files (RPM package changes and RPM repository mapping) attached to the Knowledge Base article Data required by the Leapp utility for an in-place upgrade from RHEL 7 to RHEL 8 and place these files in the /etc/leapp/files/ directory.

4. Update your Red Hat subscription:

   • If your undercloud uses the Red Hat Customer Portal for registration, refresh your current subscription to obtain access to the Red Hat Enterprise Linux 8.2 content:

     $ sudo subscription-manager refresh

   • If your undercloud uses Red Hat Satellite Server for registration, re-register the undercloud to a content view associated with your Red Hat OpenStack Platform (RHOSP)16.1 activation key.

     $ sudo subscription-manager register --force --org ORG --activationkey ACTIVATION_KEY

     Note

     The content view that you create for Red Hat OpenStack Platform 16.1 must contain content for Red Hat Enterprise Linux 8.2.

5. Red Hat OpenStack Platform 16.1 uses a newer version of Open vSwitch`. Substitute the Open vSwitch version through the to_remove and to_install transaction files:

   $ echo 'openvswitch2.11' | sudo tee -a /etc/leapp/transaction/to_remove
   $ echo 'openvswitch2.13' | sudo tee -a /etc/leapp/transaction/to_install

6. Retain the Red Hat Ceph Storage 3 version of ceph-ansible through the upgrade with the to_keep transaction file:

   $ echo 'ceph-ansible' | sudo tee -a /etc/leapp/transaction/to_keep

7. Adjust the kernel modules that are no longer supported in RHEL 8:

   $ if [ -f /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/kernel/checkkerneldrivers/files/removed_drivers.txt ]; then
       for module in pata_acpi floppy; do
           sudo sed -i "/^${module}$/d" /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/kernel/checkkerneldrivers/files/removed_drivers.txt
       done
   else
       for module in pata_acpi floppy; do
           jq ". | del(.data[] | select(.driver_name == \"${module}\"))" /etc/leapp/files/device_driver_deprecation_data.json | sudo tee /etc/leapp/files/device_driver_deprecation_data.json_modified
           mv /etc/leapp/files/device_driver_deprecation_data.json_modified /etc/leapp/files/device_driver_deprecation_data.json
       done
   fi

8. Run the leapp answer command and specify the Leapp answer to remove the pam_pkcs11 module:

   $ sudo leapp answer --add --section remove_pam_pkcs11_module_check.confirm=True

9. Optional: If your environment is deployed with a TLS-Everywhere architecture and it uses the deprecated authconfig utility to configure authentication on your system, configure your RHEL 8 system with the authselect utility:

   $ sudo leapp answer --add --section authselect_check.confirm=True

   For more information about authentication configuration during the Leapp upgrade process, see Known issues in Upgrading from RHEL 7 to RHEL 8.

10. Set the LEAPP_DEVEL_TARGET_RELEASE and LEAPP_UNSUPPORTED environment variables to specify the RHEL 8 minor version that you want to upgrade to. For RHOSP 16.1, you must set the RHEL 8 minor version to 8.2:

    $ export LEAPP_UNSUPPORTED=1
    $ export LEAPP_DEVEL_TARGET_RELEASE=8.2

    You must use the LEAPP_UNSUPPORTED environment variable every time you use a environment variable with the LEAPP_DEVEL prefix.

11. Remove the persistent network names actor from the Leapp process:

    Note

    If you do not rename the network interface names before you perform the Leapp upgrade process, the interface names might change after the upgrade to RHEL 8.2 is complete. For more information about renaming the network interface names, see Section 4.3, “Using predictable NIC names for the undercloud node”.

    $ sudo rm -f /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/persistentnetnamesdisable/actor.py

12. Start the Leapp upgrade process:

    $ sudo -E leapp upgrade --debug --enablerepo rhel-8-for-x86_64-baseos-tus-rpms --enablerepo rhel-8-for-x86_64-appstream-tus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms --enablerepo ansible-2.9-for-rhel-8-x86_64-rpms

    Use the --enablerepo option to set the repositories that you want to enable during the Leapp upgrade process. You must include these repositories to facilitate the Red Hat OpenStack Platform 16.1 transition, especially with the newer version of Open vSwitch.

13. Wait for the leapp upgrade command to successfully complete.

14. Create an empty .autorelabel file in your root directory:

    $ sudo touch /.autorelabel

    After a reboot, SELinux detects this file and automatically relabels the file system.

15. Reboot the undercloud:

    $ sudo reboot

16. Remove the Leapp packages from the transaction exclusion that is defined in the DNF configuration:

    $ sudo dnf config-manager --save --setopt exclude=''

Procedure

1. Log in to the undercloud as the stack user.

2. Lock the undercloud to a specific version with the subscription-manager release command:

   $ sudo subscription-manager release --set=8.2

Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms --enable=advanced-virt-for-rhel-8-x86_64-eus-rpms

   These repositories contain packages that the director installation requires.

3. Set the container-tools repository module to version 2.0:

   [stack@director ~]$ sudo dnf module reset container-tools
   [stack@director ~]$ sudo dnf module enable -y container-tools:2.0

4. Synchronize the operating system to ensure that your system packages match the operating system version:

   [stack@director ~]$ sudo dnf distro-sync -y
   [stack@director ~]$ sudo reboot

Procedure

1. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ sudo openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means that director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. Director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Modify the containers-prepare-parameter.yaml to suit your requirements.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the containers-prepare-parameter.yaml file.

3. Add the transitional container parameters to set in the ContainerImagePrepare parameter. Set the parameters in one of the following ways depending on your type of deployment.

   • If your deployment uses a Red Hat Ceph Storage cluster that was deployed using director, add the following parameters:

     parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
         set:
           ...
           name_prefix_stein: openstack-
           name_suffix_stein: ''
           namespace_stein: registry.redhat.io/rhosp15-rhel8
           tag_stein: 15.0
           ceph3_namespace: registry.redhat.io/rhceph
           ceph3_tag: latest
           ceph3_image: rhceph-3-rhel7
           ...

     • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
     • The ceph3_* parameters define the current Red Hat Ceph Storage container images that the overcloud uses. The overcloud requires both the ceph3_* and ceph_* parameters for the transition from Red Hat Ceph Storage 3 to 4.
     • If you use Red Hat Satellite Server for container image storage, set the namespaces to the image locations on your Red Hat Satellite Server.

   • If your deployment uses an external Ceph Storage cluster, add the following parameters:

     parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
         set:
           ...
           name_prefix_stein: openstack-
           name_suffix_stein: ''
           namespace_stein: registry.redhat.io/rhosp15-rhel8
           tag_stein: 15.0
           ceph_namespace: registry.redhat.io/rhceph
           ceph_tag: latest
           ceph_image: rhceph-4-rhel8
           ...

     • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
     • The ceph_* parameters define the current Red Hat Ceph Storage 4 container images that the overcloud uses.
     • If you use Red Hat Satellite Server for container image storage, set the namespaces to the image locations on your Red Hat Satellite Server.

4. Change the neutron_driver parameter to openvswitch:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ...
         neutron_driver: openvswitch
         ...

   The upgrade retains Open vSwitch compatibility throughout the process. After you complete the upgrade to Red Hat OpenStack Platform 16.1, you migrate the overcloud from Open vSwitch to Open Virtual Network (OVN).

5. Save the containers-prepare-parameter.yaml file.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the undercloud.conf file.

3. Add the following parameter to the DEFAULT section in the file:

   container_images_file = /home/stack/containers-prepare-parameter.yaml

   This parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.

4. Check the generate_service_certificate parameter. The default for this parameter changes from false to true, which enables SSL/TLS on your undercloud, during the upgrade.
5. Check the local_interface parameter if you have migrated to a predictable NIC naming convention.
6. If you set the masquerade_network parameter in Red Hat OpenStack Platform 13, remove this parameter and set masquerade = true for each subnet.
7. Check all other parameters in the file for any changes.
8. Save the file.

Procedure

1. Run the following command to upgrade the director on the undercloud:

   $ openstack undercloud upgrade

   This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

   Note

   The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the -y option:

   $ openstack undercloud upgrade -y

2. The script also starts all OpenStack Platform service containers on the undercloud automatically. You manage each service through a systemd resource. Check the systemd resources:

   $ sudo systemctl list-units "tripleo_*"

   Each systemd service controls a container. Check the enabled containers using the following command:

   $ sudo podman ps

3. The script adds the stack user to the docker group to ensure that the stack user has access to container management commands. Refresh the stack user permissions with the following command:

   $ exec su -l stack

   The command prompts you to log in again. Enter the stack user password.

4. To initialize the stack user to use the command line tools, run the following command:

   $ source ~/stackrc

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

   (undercloud) $

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Create a static inventory file of all nodes:

   $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml --stack STACK_NAME

   If you are not using the default overcloud stack name, set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

4. To execute Ansible playbooks on your environment, run the ansible-playbook command and include the full path of the dynamic inventory tool using the -i option. For example:

   (undercloud) $ ansible-playbook -i ~/inventory.yaml PLAYBOOK

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group pre-upgrade option and include the /usr/libexec/platform-python python runtime environment:

   $ openstack tripleo validator run --group pre-upgrade --python-interpreter /usr/libexec/platform-python -i inventory.yaml

3. Review the results of the validation report. To view detailed output from a specific validation, run the openstack tripleo validator show run --full command against the UUID of the specific validation from the report:

   $ openstack tripleo validator show run  --full <UUID>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to disable fencing:

   $ ssh heat-admin@CONTROLLER_IP "sudo pcs property set stonith-enabled=false"

4. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the upgrade process.

Procedure

1. Select an environment file and the check if it has an ExtraConfig parameter:

   $ grep ExtraConfig ~/templates/custom-config.yaml

2. If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig) in the chosen file, check the full parameter structure in that file.

3. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value, it might have been been replaced with a parameter with an actual Puppet class. For example:

   parameter_defaults:
     ExtraConfig:
       neutron::config::dhcp_agent_config:
         'DEFAULT/dnsmasq_local_resolv':
           value: 'true'

4. Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:

   $ grep dnsmasq_local_resolv

   If so, change to the new interface.

5. The following are examples to demonstrate the change in syntax:

   • Example 1:

     parameter_defaults:
       ExtraConfig:
         neutron::config::dhcp_agent_config:
           'DEFAULT/dnsmasq_local_resolv':
             value: 'true'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         neutron::agents::dhcp::dnsmasq_local_resolv: true

   • Example 2:

     parameter_defaults:
       ExtraConfig:
         ceilometer::config::ceilometer_config:
           'oslo_messaging_rabbit/rabbit_qos_prefetch_count':
             value: '32'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'

Procedure

1. Log in to the undercloud as the stack user.

2. Create an environment file called upgrades-environment.yaml in your templates directory:

   $ touch templates/upgrades-environment.yaml

3. Edit the file and add the following mandatory content:

   parameter_defaults:
     UpgradeLeappDevelSkip: "LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_TARGET_RELEASE=8.2"
     LeappInitCommand: |
       for module in pata_acpi floppy; do sudo sed -i "/^${module}$/d" /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/kernel/checkkerneldrivers/files/removed_drivers.txt; done
       sudo rm -f /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/persistentnetnamesdisable/actor.py
       sudo yum -y remove mariadb-server* || true
     UpgradeInitCommand: "sudo dnf config-manager --save --setopt exclude=''"

   • UpgradeLeappDevelSkip skips Leapp checks and sets environment variables when Leapp runs.
   • LeappInitCommand passes commands or script snippets that run on each of the overcloud nodes, and prepares the nodes for the Leapp upgrade.
   • UpgradeInitCommand passes commands or script snippets that run on each of the overcloud nodes. The dnf config-manager --save --setopt exclude='' command removes Leapp packages from DNF exclusion so that automatic removal of python2 packages is successful.

4. Optional: If you use TLS-Everywhere in your environment and want to migrate from authconfig to authselect, set the authselect_check.confirm parameter to True to avoid bug BZ#1978228 - OSP13→16.2 Leapp upgrade failed with TLSEverywhere:

   parameter_defaults:
     LeappInitCommand: |
       sudo leapp answer --section authselect_check.confirm=True --add

   Otherwise, set the value to False.

   Note

   Use the | syntax to pass multiple commands to the LeappInitCommand parameter:

   parameter-defaults:
     LeappInitCommand: |
       <command_1>
       <command_2>

5. Save the upgrades-environment.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Create a static inventory file of all the nodes in your environment:

   $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml --stack STACK_NAME

   If you are not using the default overcloud stack name, set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

4. To copy the leapp data to the overcloud nodes, run the following synchronize Ansible command:

   $ ansible -i ~/inventory.yaml --become -m synchronize -a "src=/etc/leapp/files dest=/etc/leapp/" overcloud

Procedure

1. Log in to the undercloud as the stack user.

2. Run the playbook-nics.yaml playbook on all overcloud nodes:

   $ ansible-playbook -i ~/inventory.yaml playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -i ~/inventory.yaml -e prefix="mynic" playbook-nics.yaml

   The NIC changes are only applied after the Leapp upgrade process completes and the node is rebooted.

Procedure

1. Search for custom environment files that use composable services. You usually store custom environment files in the /home/stack/templates directory:

   $ cd ~/templates/
   $ grep "OS::TripleO::Services" *

   In this scenario, one of the files shows an outdated mapping:

     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mgr.yaml

2. Identify the new ceph-mgr.yaml location in /usr/share/openstack-tripleo-heat-templates/. This file is now located in the `deployment/ceph-ansible' directory:

   $ find /usr/share/openstack-tripleo-heat-templates/ -name ceph-mgr.yaml
   /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

3. Edit the service in the custom environment file:

   resource_registry:
     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

   Save the file.

Procedure

1. Log in to the undercloud as the stack user.

2. Obtain the control plane host name on the undercloud:

   $ sudo hiera container_image_prepare_node_names
   ["undercloud.ctlplane.localdomain"]

3. Edit the containers-prepare-parameter.yaml file and add the DockerInsecureRegistryAddress parameter with a YAML list that contains the control plane host name of the undercloud and the IP address of the undercloud on the provisioning network:

   parameter_defaults:
     DockerInsecureRegistryAddress:
     - undercloud.ctlplane.localdomain:8787
     - 192.168.24.1:8787
     ...

   You must also append the port number of the overcloud registry to the host name and IP address values. The port number is 8787.

Procedure

1. Log in to the undercloud as the stack user.

2. Set the Compute naming format:

   • If you use a custom roles_data file, edit the custom roles_data file and set the HostnameFormatDefault parameter for the Compute role:

     - name: Compute
       …​
       HostnameFormatDefault: '%stackname%-compute-%index%'
       …​

     Save the custom roles_data file.

   • If you use the default roles_data file in openstack-tripleo-heat-templates, set the naming format in an environment file. Edit the environment file with your node counts and flavors, which is usually named node-info.yaml. Add the ComputeHostnameFormat parameter to the parameter_defaults section:

     parameter_defaults:
       …​
       ComputeHostnameFormat: '%stackname%-compute-%index%'
       …​

     Save the node-info.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Open an environment file.

4. Add the following configuration to your environment file to specify that the instance serial number is based on the serial number of the host:

   parameter_defaults:
     <Role>ExtraConfig:
       nova::config::nova_config:
         libvirt/sysinfo_serial:
           value: auto

5. Save the updates to your environment file, and add the file to your overcloud upgrade and deployment commands.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Edit your custom overcloud SSL/TLS public endpoint file, which is usually named ~/templates/enable-tls.yaml.

4. Remove the NodeTLSData resource from the `resource_registry:

   resource_registry:
     OS::TripleO::NodeTLSData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/tls-cert-inject.yaml
     …​

   The overcloud deployment uses a new service in HAProxy to determine if SSL/TLS is enabled.

   Note

   If this is the only resource in the resource_registry section of the enable-tls.yaml file, remove the complete resource_registry section.

5. Save the SSL/TLS public endpoint file file.

Procedure

1. Edit the post-configuration template.

2. Under the parameters section, add the EndpointMap parameter and its sub-parameters:

   parameters:
     servers:
       type: json
     nameserver_ip:
       type: string
     DeployIdentifier:
       type: string
     EndpointMap:
       default: {}
       type: json

3. Save the template.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file named templates/rhsm.yml to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-8-for-x86_64-baseos-tus-rpms
         - rhel-8-for-x86_64-appstream-tus-rpms
         - rhel-8-for-x86_64-highavailability-tus-rpms
         …​
       rhsm_username: "myusername"
       rhsm_password: "p@55w0rd!"
       rhsm_org_id: "1234567"
       rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
       rhsm_method: "portal"
       rhsm_release: 8.2

   • The resource_registry section associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.
   • The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.
3. Save the environment file.

Procedure

1. Create an environment file named templates/rhsm.yml to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     ControllerParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-tus-rpms
           - rhel-8-for-x86_64-appstream-tus-rpms
           - rhel-8-for-x86_64-highavailability-tus-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - advanced-virt-for-rhel-8-x86_64-eus-rpms
           - openstack-16.1-for-rhel-8-x86_64-rpms
           - fast-datapath-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "55d251f1490556f3e75aa37e89e10ce5"
         rhsm_method: "portal"
         rhsm_release: 8.2
     ComputeParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-tus-rpms
           - rhel-8-for-x86_64-appstream-tus-rpms
           - rhel-8-for-x86_64-highavailability-tus-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - advanced-virt-for-rhel-8-x86_64-eus-rpms
           - openstack-16.1-for-rhel-8-x86_64-rpms
           - fast-datapath-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "55d251f1490556f3e75aa37e89e10ce5"
         rhsm_method: "portal"
         rhsm_release: 8.2
     CephStorageParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-tus-rpms
           - rhel-8-for-x86_64-appstream-tus-rpms
           - rhel-8-for-x86_64-highavailability-tus-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - openstack-16.1-deployment-tools-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "68790a7aa2dc9dc50a9bc39fabc55e0d"
         rhsm_method: "portal"
         rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The ControllerParameters, ComputeParameters, and CephStorageParameters parameters each use a separate RhsmVars parameter to pass subscription details to their respective roles.

   Note

   Set the RhsmVars parameter within the CephStorageParameters parameter to use a Red Hat Ceph Storage subscription and repositories specific to Ceph Storage. Ensure the rhsm_repos parameter contains the standard Red Hat Enterprise Linux repositories instead of the Extended Update Support (EUS) repositories that Controller and Compute nodes require.

3. Save the environment file.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file named templates/rhsm.yml to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_activation_key: "myactivationkey"
       rhsm_method: "satellite"
       rhsm_org_id: "ACME"
       rhsm_server_hostname: "satellite.example.com"
       rhsm_baseurl: "https://satellite.example.com/pulp/repos"
       rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

3. Save the environment file.

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-satellite.yaml and paste the following content in the file:

   - name: Pre-install leapp
     hosts: overcloud
     become: yes
     tasks:
       - name: Pre-install leapp
         yum:
           name:
             - leapp
             - leapp-repository
           state: installed
       - name: Remove katello-host-tools-fact-plugin
         yum:
           name:
             - katello-host-tools-fact-plugin
           state: removed
       - name: Register system
         redhat_subscription:
           activationkey: "osp16-key"
           org_id: "ACME"
           server_hostname: "satellite.example.com"
           rhsm_baseurl: "https://satellite.example.com/pulp/repos"
           force_register: True

   Modify the redhat_subscription variables to suit your Satellite Server.

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-satellite.yaml

1. Upgrade the undercloud but retain version 3 of ceph-ansible
2. Start the overcloud upgrade

3. Perform the following tasks for each node that hosts Ceph Storage containerized services:

   1. Migrate the Ceph Storage containerized services to Podman
   2. Upgrade the operating system
   3. Upgrade the OpenStack Platform services, which relaunches Ceph Storage version 3 containerized services
4. Complete the overcloud upgrade
5. Upgrade ceph-ansible to version 4 on the undercloud
6. Upgrade to Red Hat Ceph Storage 4 on the overcloud

Procedure

1. Log in to the undercloud as the stack user.

2. Run the dnf command to check the version of the ceph-ansible package:

   $ sudo dnf info ceph-ansible

   The command output shows version 3 of the ceph-ansible package:

   Installed Packages
   Name         : ceph-ansible
   Version      : 3.xx.xx.xx
   ...

Procedure

1. Log in to the undercloud as the stack user.

2. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named ceph-config.yaml and you can find it in your templates directory:

   $ vi /home/stack/templates/ceph-config.yaml

3. Add the CephAnsibleRepo parameter to the parameter_defaults section:

   parameter_defaults:
     ...
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
     ...

   CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.

4. Save the ceph-config.yaml file.

Procedure

1. Log in to the node that is running the ceph-mon service. This node is usually a Controller node or a standalone Ceph Monitor node.

2. Enter the following command to view the status of the Ceph cluster:

   $ docker exec ceph-mon-$HOSTNAME ceph -s

3. Confirm that the health status of the cluster is HEALTH_OK and that all of the OSDs are up.

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to the undercloud as the stack user.

2. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named ceph-config.yaml and you can find it in your templates directory:

   $ vi /home/stack/templates/ceph-config.yaml

3. Add the CephAnsibleRepo parameter to the parameter_defaults section:

   parameter_defaults:
     ...
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
     ...

   CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.

4. Save the ceph-config.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. On the undercloud, create a file called update-nic-templates.sh and include the following content in the file:

   #!/bin/bash
   STACK_NAME="overcloud"
   ROLES_DATA="/usr/share/openstack-tripleo-heat-templates/roles_data.yaml"
   NETWORK_DATA="/usr/share/openstack-tripleo-heat-templates/network_data.yaml"
   NIC_CONFIG_LINES=$(openstack stack environment show $STACK_NAME | grep "::Net::SoftwareConfig" | sed -E 's/ *OS::TripleO::// ; s/::Net::SoftwareConfig:// ; s/ http.*user-files/ /')
   echo "$NIC_CONFIG_LINES" | while read LINE; do
       ROLE=$(echo "$LINE" | awk '{print $1;}')
       NIC_CONFIG=$(echo "$LINE" | awk '{print $2;}')
   
       if [ -f "$NIC_CONFIG" ]; then
           echo "Updating template for $ROLE role."
           python3 /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
               --tht-dir /usr/share/openstack-tripleo-heat-templates \
               --roles-data $ROLES_DATA \
               --network-data $NETWORK_DATA \
               --role-name "$ROLE" \
               --discard-comments yes \
               --template "$NIC_CONFIG"
       else
           echo "No NIC template detected for $ROLE role. Skipping $ROLE role."
       fi
   done

   • If you use a custom overcloud name, Set the STACK_NAME variable to the name of your overcloud. The default name for an overcloud stack is overcloud.
   • If you use a custom roles_data file, set the ROLES_DATA variable to the location of the custom file. If you use the default roles_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
   • If you use a custom network_data file, set the NETWORK_DATA variable to the location of the custom file. If you use the default network_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/network_data.yaml
   • Run /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py -h to see a list of options to add to the script.

4. Add executable permissions to the script:

   $ chmod +x update-nic-templates.sh

5. Optional: If you use a spine-leaf network topology for your RHOSP environment, check the roles_data.yaml file and ensure that it uses the correct role names for the NIC templates for your deployment. The script uses the value of the deprecated_nic_config_name parameter in the roles_data.yaml file.

6. Run the script:

   $ ./update-nic-templates.sh

   The script saves a copy of each custom NIC template and updates each template with the missing parameters. The script also skips any roles that do not have a custom template:

   No NIC template detected for BlockStorage role. Skipping BlockStorage role.
   Updating template for CephStorage role.
   The original template was saved as: /home/stack/templates/custom-nics/ceph-storage.yaml.20200903144835
   The update template was saved as: /home/stack/templates/custom-nics/ceph-storage.yaml
   Updating template for Compute role.
   The original template was saved as: /home/stack/templates/custom-nics/compute.yaml.20200903144838
   The update template was saved as: /home/stack/templates/custom-nics/compute.yaml
   Updating template for Controller role.
   The original template was saved as: /home/stack/templates/custom-nics/controller.yaml.20200903144841
   The update template was saved as: /home/stack/templates/custom-nics/controller.yaml
   No NIC template detected for ObjectStorage role. Skipping ObjectStorage role.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on the undercloud node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

3. Upgrade the bootstrap Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack <stack_name> --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      Replace <stack_name> with the name of your stack.

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

   6. Verify that after the upgrade, the new Pacemaker cluster is started and that the control plane services such as galera, rabbit, haproxy, and redis are running:

      $ sudo pcs status

4. Upgrade the next Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Optional: If you use a Ceph subscription and have configured director to use the overcloud-minimal image for Ceph storage nodes, you must complete the following steps:

      1. Log in to the node and unset the Red Hat Enterprise Linux (RHEL) minor release version:

         $ sudo subscription-manager release --unset

      2. On the node, perform a system update:

         $ sudo dnf -y update

      3. Reboot the node:

         $ sudo reboot

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on the undercloud node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

3. Upgrade the bootstrap Controller node:

   1. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   2. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   3. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

   5. Verify that after the upgrade, the new Pacemaker cluster is started and that the control plane services such as galera, rabbit, haproxy, and redis are running:

      $ sudo pcs status

4. Upgrade the next Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. On the undercloud node, run the following command to identify the bootstrap Controller node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

4. Upgrade the overcloud-controller-0, overcloud-database-0, overcloud-networker-0, and overcloud-ceph-0 control plane nodes:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack <stack_name> --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0,overcloud-database-0,overcloud-networker-0,overcloud-ceph-0

      Replace <stack_name> with the name of your stack.

      This command performs the following actions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected nodes using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for the leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 &
      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-database-0 &
      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-networker-0 &
      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-ceph-0 &

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-database-0,overcloud-networker-0,overcloud-ceph-0 --playbook all

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

   6. Optional: On the bootstrap Contoller node, verify that after the upgrade, the new Pacemaker cluster is started and that the control plane services such as galera, rabbit, haproxy, and redis are running:

      $ sudo pcs status

5. Upgrade the overcloud-controller-1, overcloud-database-1, overcloud-networker-1, and overcloud-ceph-1 control plane nodes:

   1. Log in to the overcloud-controller-1 node and verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1,overcloud-database-1,overcloud-networker-1,overcloud-ceph-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected nodes using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1,overcloud-database-1,overcloud-networker-1,overcloud-ceph-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-database-0,overcloud-database-1,overcloud-networker-0,overcloud-networker-1,overcloud-ceph-0,overcloud-ceph-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap nodes in the --limit option.

6. Upgrade the overcloud-controller-2, overcloud-database-2, overcloud-networker-2, and overcloud-ceph-2 control plane nodes:

   1. Log in to the overcloud-controller-2 node and verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2,overcloud-database-2,overcloud-networker-2,overcloud-ceph-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected nodes using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2,overcloud-database-2,overcloud-networker-2,overcloud-ceph-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2,overcloud-database-0,overcloud-database-1,overcloud-database-2,overcloud-networker-0,overcloud-networker-1,overcloud-networker-2,overcloud-ceph-0,overcloud-ceph-1,overcloud-ceph-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all control plane nodes in the --limit option.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

4. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run -y --stack STACK NAME --tags system_upgrade --limit 'Compute[0:19]' > upgrade-compute-00-19.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --tags system_upgrade --limit 'Compute[20:29]' > upgrade-compute-20-29.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --tags system_upgrade --limit 'Compute[40:59]' > update-compute-40-59.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --tags system_upgrade --limit 'Compute[60:79]' > update-compute-60-79.log 2>&1 &

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

5. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run -y --stack STACK NAME --limit 'Compute[0:19]' > upgrade-compute-00-19.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --limit 'Compute[20:29]' > upgrade-compute-20-29.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --limit 'Compute[40:59]' > update-compute-40-59.log 2>&1 &
   $ openstack overcloud upgrade run -y --stack STACK NAME --limit 'Compute[60:79]' > update-compute-60-79.log 2>&1 &

   This command performs the Red Hat OpenStack Platform upgrade.

6. Optional: To upgrade selected Compute nodes, use the --limit option with a comma-separated list of nodes that you want to upgrade. The following example upgrades the overcloud-compute-0, overcloud-compute-1, overcloud-compute-2 nodes in parallel.

   1. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   2. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap node by running the following command on the undercloud node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

3. Upgrade the bootstrap node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack <stack_name> --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      Replace <stack_name> with the name of your stack.

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

4. Upgrade each Pacemaker-based node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-database-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-database-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-database-0

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include any previously upgraded node in the --limit option.

5. Repeat the upgrade process on each Pacemaker-based node until you have upgraded all Pacemaker-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-networker-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

3. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-networker-0

   This command performs the Red Hat OpenStack Platform upgrade.

4. Repeat the upgrade process on each node until you have upgraded all Controller-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Select a Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-0

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-1

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-2

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Optional: If you use a Ceph subscription and have configured director to use the overcloud-minimal image for Ceph storage nodes, you must complete the following steps:

      1. Log in to the node and unset the Red Hat Enterprise Linux (RHEL) minor release version:

         $ sudo subscription-manager release --unset

      2. On the node, perform a system update:

         $ sudo dnf -y update

      3. Reboot the node:

         $ sudo reboot

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on the undercloud node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

3. Upgrade the bootstrap Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack <stack_name> --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      Replace <stack_name> with the name of your stack.

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

   6. Verify that after the upgrade, the new Pacemaker cluster is started and that the control plane services such as galera, rabbit, haproxy, and redis are running:

      $ sudo pcs status

4. Upgrade the next Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.
3. Log out of the node with Ceph MON services and return to the undercloud.

4. Run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

   This command performs the following functions:

   • Changes the systemd units that control the Ceph Storage containers to use Podman management.
   • Limits actions to the selected Ceph Storage node using the ceph_ansible_limit variable.

   This step is a preliminary measure to prepare the Ceph Storage services for the leapp upgrade.

5. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

6. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-computehci-0

   This command performs the Red Hat OpenStack Platform upgrade.

7. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2

   Then perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph

3. Wait until the Ceph Storage upgrade completes.

Procedure

1. Log in as the heat-admin user on a node with Ceph MON containers, such as Controller nodes or standalone Ceph MON nodes. For example, in a standard overcloud deployment, overcloud-controller-1 uses Ceph MON containers.

2. Query the Ceph cluster to see what driver is in use by the OSDs:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]#

3. If any line returns "objectstore": "filestore", that node requires OSD migration.

Procedure

1. Ensure that osd_objectstore in the CephAnsibleDisksConfig parameter does not default to filestore. If the osd_objectstore parameter is present in any of your custom heat environment files, you must define the value bluestore explicitly or remove it:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
      osd_objectstore: bluestore

   Note

   If you have any specific FileStore configuration with, for example, journals, ensure that you update the configuration accordingly. For more information about advanced configurations, see Mapping the Ceph Storage Node Disk Layout in the Deploying an overcloud with containerized Red Hat Ceph guide.

2. Log in to the undercloud as the stack user.

3. Enter the openstack overcloud external-upgrade run command with the ceph_fstobs tag. Replace <NODE_NAME> with the name of the Ceph OSD node you want to upgrade. You can use the openstack server list command to find the node name.

   [stack@undercloud ~] $ openstack overcloud external-upgrade run --tags ceph_fstobs -e ceph_ansible_limit=<NODE_NAME> | tee oc-fstobs.log

4. Log in to a node that has Ceph MON services and query the Ceph cluster to check the status of the OSDs of the node you have upgraded. Before you can start the migration of the next OSD node, you must ensure that the one you have upgraded is successfully migrated:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c '.[] | select(.hostname == "<NODE_NAME>") | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]# exit

   Replace <NODE_NAME> with the name of the node that was migrated. If the result shows that the OSDs use BlueStore, its migration is successful.

5. Optional: To view additional details about a specific OSD, enter the following command:

   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph osd metadata <ID>"

   Replace <ID> with the ID of the OSD you want to query.

6. Before you can start the migration process on the next node, you must wait for the cluster to synchronize.

   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c  "ceph  -s"

   Review the command output and ensure that the health of the cluster is `HEALTH_OK` and the PGs are in the `active+clean` state.

7. Before you can start the migration process on the next node, you must wait for the cluster rebalancing process to complete. To follow the status, run the following command:

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<NODE_NAME> ceph -w

   Replace <NODE_NAME> with the name of the node that was migrated.

8. Repeat the migration process for each node in the storage cluster.

Procedure

1. Log in as the heat-admin user to a ceph-mon container that is hosted on one of the Controller nodes.

2. Query the Ceph cluster:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]#

Procedure

1. Remove the unnecessary packages

   $ sudo dnf -y remove --exclude=python-pycadf-common python2*

2. Remove the content from the /httpboot and /tftpboot directories that includes old images used in Red Hat OpenStack 13:

   $ sudo rm -rf /httpboot /tftpboot

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group post-upgrade option:

   $ openstack tripleo validator run --group post-upgrade

3. Review the results of the validation report. To view detailed output from a specific validation, run the openstack tripleo validator show run --full command against the UUID of the specific validation from the report:

   $ openstack tripleo validator show run --full <UUID>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Install the packages containing the overcloud QCOW2 archives:

   $ sudo dnf install rhosp-director-images rhosp-director-images-ipa

4. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

5. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done
   $ cd ~

6. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

7. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)