Framework for upgrades (16.2 to 17.1)

Procedure

1. Check the status of the OVN cluster resources and fix any issues:

   $ sudo pcs status

   Note

   If you need help resolving any issues, contact Red Hat Support before you proceed with the upgrade.

2. Check in the Failed Resource Actions section of the output of the OVN cluster resources status for the following error:

   Failed Resource Actions:
     * ovndb_servers_monitor_30000 on ovn-dbs-bundle-1 'not running' (7): call=24, status='complete', exitreason='', last-rc-change='2025-08-08 09:21:30Z', queued=0ms, exec=159ms

3. If you see the error, check the status of the ovndb_servers resource on controller-0:

   $ sudo podman exec <ovn-dbs container> ovs-appctl -t /var/run/openvswitch/ovnnb_db.ctl ovsdb-server/sync-status

   • Replace <ovn-dbs container> with the name of the ovn-dbs container that you want to check.

4. If the output includes a reference to the IP address, 192.0.2.254, instead of the virtual IP address, you must restart the ovn-dbs-bundle on the overcloud Controller node:

   $ sudo pcs resource restart ovn-dbs-bundle

   192.0.2.254 is a fallback IP address. Its presence in the output indicates that there is an issue.

Procedure

1. Confirm that all bare-metal provisioned nodes are in the ACTIVE state:

   $ openstack baremetal node list

2. If any nodes are in the MAINTENANCE state, identify and troubleshoot the root cause of the nodes that are in MAINTENANCE by running the following command and checking the last_error field:

   $ openstack baremetal node show <node_uuid>

   • Replace <node_uuid> with the UUID of the node.

3. Unset the MAINTENANCE state:

    $ openstack baremetal node maintenance unset <node_uuid>

   Wait three to five minutes to see if the node returns to the MAINTENANCE state.

   Important

   If you are unable to remove the nodes from MAINTENANCE, contact Red Hat Support.

Procedure

1. Open the configuration file that includes the following service:

   OS::TripleO::Services::OVNDBs: deployment/ovn/ovn-dbs-pacemaker-puppet.yaml

2. Replace OS::TripleO::Services::OVNDBs: deployment/ovn/ovn-dbs-pacemaker-puppet.yaml with the following service:

   OS::TripleO::Services::OVNDBs: deployment/ovn/ovn-dbs-cluster-ansible.yaml

3. Save the file.

Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux (RHEL) repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos \
   --enable=rhel-8-for-x86_64-baseos-aus-rpms \
   --enable=rhel-8-for-x86_64-appstream-aus-rpms \
   --enable=rhel-8-for-x86_64-highavailability-aus-rpms \
   --enable=openstack-17.1-for-rhel-8-x86_64-rpms \
   --enable=fast-datapath-for-rhel-8-x86_64-rpms

3. Switch the container-tools module version to RHEL 8 on all nodes:

   [stack@director ~]$ sudo dnf -y module switch-to container-tools:rhel8

4. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Confirm that you can ping the overcloud nodes:

   $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml --stack <stack> --ansible_ssh_user heat-admin
   $ ansible -i ~/inventory.yaml all -m ping

   • Replace <stack> with the name of the stack.

4. Adjust the permissions of the /var/lib/mistral/.ssh directory:

   $ sudo chmod +x /var/lib/mistral/.ssh/

5. Install the packages for validation:

   $ sudo dnf -y update openstack-tripleo-validations python3-validations-libs validations-common

6. Copy the inventory from mistral:

   $ sudo chown stack:stack /var/lib/mistral/.ssh/tripleo-admin-rsa
   $ sudo cat /var/lib/mistral/<stack>/tripleo-ansible-inventory.yaml > inventory.yaml

7. Run the validation:

   $ validation run -i inventory.yaml --group pre-upgrade

8. Review the script output to determine which validations succeed and fail:

   === Running validation: "check-ftype" ===
   
   Success! The validation passed for all hosts:
   * undercloud

9. Remove the inventory.yaml file:

   $ rm inventory.yaml

Procedure

1. Log in to the undercloud host as the stack user.

2. Optional: Back up the 16.2 containers-prepare-parameter.yaml file:

   $ cp containers-prepare-parameter.yaml \
     containers-prepare-parameter.yaml.orig

3. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means that director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. Director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

4. Modify the containers-prepare-parameter.yaml to suit your requirements. For more information about container image parameters, see Container image preparation parameters.

5. If your deployment includes Red Hat Ceph Storage, update the Red Hat Ceph Storage container image parameters in the containers-prepare-parameter.yaml file for the version of Red Hat Ceph Storage that your deployment uses:

   ceph_namespace: registry.redhat.io/rhceph
   ceph_image: <ceph_image_file>
   ceph_tag: latest
   ceph_grafana_image: <grafana_image_file>
   ceph_grafana_namespace: registry.redhat.io/rhceph
   ceph_grafana_tag: latest

   • Replace <ceph_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • If you use director-deployed Red Hat Ceph Storage, replace <ceph_image_file> with rhceph-5-rhel8.
     • If you use external Red Hat Ceph Storage, replace <ceph_image_file> with the same Ceph image that your Red Hat Ceph Storage environment uses. For example, for a Red Hat Ceph Storage 6 image, use rhceph-6-rhel9.

   • Replace <grafana_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • If you use director-deployed Red Hat Ceph Storage, replace <grafana_image_file> with rhceph-5-dashboard-rhel8.
     • If you use external Red Hat Ceph Storage, replace <grafana_image_file> with the same Ceph image that your Red Hat Ceph Storage environment uses. For example, for a Red Hat Ceph Storage 6 image, use rhceph-6-dashboard-rhel9.

Procedure

1. Log in to your undercloud host as the stack user.

2. Create a file called skip_rhel_release.yaml and set the SkipRhelEnforcement parameter to true:

   parameter_defaults:
     SkipRhelEnforcement: true

3. Open the undercloud.conf file, and add the container_images_file parameter to the DEFAULT section in the file:

   container_images_file = /home/stack/containers-prepare-parameter.yaml

   • The container_images_file parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.

4. Add the custom_env_files parameter to the DEFAULT section in the undercloud.conf file. The custom_env_files parameter defines the location of the skip_rhel_release.yaml file that is required for the upgrade:

   custom_env_files = /home/stack/skip_rhel_release.yaml

   • Add any additional custom environment files to the custom_env_files parameter, separated by a comma. Ensure that any existing files in the parameter are included in the list. For example:

     custom_env_files = /home/stack/custom-undercloud-params.yaml,/home/stack/skip_rhel_release.yaml

5. Check all other parameters in the file for any changes.
6. Save the file.

Procedure

1. Confirm that the tripleo_mysql.service is running:

   $ systemctl status tripleo_mysql

   Sample output

   The command shows the following output if the service is not running:

   tripleo_mysql.service - mysql container
      Loaded: loaded (/etc/systemd/system/tripleo_mysql.service; enabled; vendor preset: disabled)
      Active: inactive (dead)

2. If the service is not running, start the service:

   $ sudo systemctl start tripleo_mysql

3. Launch the director configuration script to upgrade director:

   $ openstack undercloud upgrade

   The director configuration script upgrades director packages and configures director services to match the settings in the undercloud.conf file. This script takes several minutes to complete.

   Note

   The director configuration script prompts for confirmation before proceeding. Bypass this confirmation by using the -y option:

   $ openstack undercloud upgrade -y

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. If you included the environments/ceph-ansible/ceph-ansible-external.yaml file in the following commands, you must replace the file with the environments/external-ceph.yaml file.

   • openstack overcloud upgrade prepare

   • openstack overcloud deploy

     For example, replace

     $ openstack overcloud deploy
     ...
     -e environments/ceph-ansible/ceph-ansible-external.yaml
     ...

     with

     $ openstack overcloud deploy
     ...
     -e environments/external-ceph.yaml
     ...

4. Create a file called ceph_params.yaml and include the following content:

   parameter_defaults:
     CephClusterFSID: <fsid>
     CephClientKey: <key>
     CephExternalMonHost: <mon ip addresses>
     CephSpecFqdn: <true/false>
     CephConfigPath: "/etc/ceph"
     DeployedCeph: false
     GrafanaPlugins: []

   • Replace <fsid> with the UUID of your Red Hat Ceph Storage cluster.
   • Replace <key> with your Ceph client key.
   • Replace <mon ip addresses> with a list of your Ceph Mon Host IPs.

   • Replace <true/false> with the value that applies to your environment.

     Note

     If your Red Hat Ceph Storage deployment includes short names, you must set the CephSpecFqdn parameter to false. If set to true, the inventory generates with both the short names and domain names, causing the Red Hat Ceph Storage upgrade to fail.

5. Include the ceph_params.yaml file in the overcloud deployment command:

   $ openstack overcloud deploy \
   ...
   -e ~/environments/ceph_params.yaml \

   Important

   Do not remove the ceph_params.yaml file after the RHOSP upgrade is complete. This file must be present in external Red Hat Ceph Storage environments. Additionally, any time you run openstack overcloud deploy, you must include the ceph_params.yaml file, for example, -e ceph_params.yaml.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. For each Controller node, log in to the Controller node and run the Pacemaker command to disable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes at /etc/hosts or /var/lib/mistral.

4. If you use SBD fencing, disable SBD fencing on the pacemaker_remote nodes:

   # pcs property set stonith-watchdog-timeout=0

   Note

   Ensure that you take note of the original value of the watchdog timer device interval. You must reset the watchdog timer device interval to its original value after you upgrade the control plane nodes. For more information, see Re-enabling fencing in the overcloud.

5. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the upgrade process.

Procedure

1. Select an environment file and the check if it has an ExtraConfig parameter:

   $ grep ExtraConfig ~/templates/custom-config.yaml

2. If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig) in the chosen file, check the full parameter structure in that file.

3. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value, it might have been been replaced with a parameter with an actual Puppet class. For example:

   parameter_defaults:
     ExtraConfig:
       neutron::config::dhcp_agent_config:
         'DEFAULT/dnsmasq_local_resolv':
           value: 'true'

4. Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:

   $ grep dnsmasq_local_resolv

   If so, change to the new interface.

5. The following are examples to demonstrate the change in syntax:

   • Example 1:

     parameter_defaults:
       ExtraConfig:
         neutron::config::dhcp_agent_config:
           'DEFAULT/dnsmasq_local_resolv':
             value: 'true'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         neutron::agents::dhcp::dnsmasq_local_resolv: true

   • Example 2:

     parameter_defaults:
       ExtraConfig:
         ceilometer::config::ceilometer_config:
           'oslo_messaging_rabbit/rabbit_qos_prefetch_count':
             value: '32'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Verify that the following files that were exported during the undercloud upgrade contain the expected configuration for the overcloud upgrade. You can find the following files in the ~/overcloud-deploy/$(<stack>) directory:

   • tripleo-<stack>-passwords.yaml
   • tripleo-<stack>-network-data.yaml
   • tripleo-<stack>-virtual-ips.yaml
   • <stack>-network-environment.yaml

   • tripleo-<stack>-baremetal-deployment.yaml

     • Replace <stack> with the name of your stack.
     • Check that the mappings between the baremetal node UUIDs, hostnames, and IP addresses in the tripleo-<stack>-baremetal-deployment.yaml file match your current overcloud. Correct the mappings if required.
     • If the tripleo-<stack>-passwords.yaml file does not exist, review the resolution in the Red Hat Knowledgebase article Create a missing overcloud password file.
   Warning

   You must address the issues in Jinja2 comments, and then remove all Jinja2 comment blocks, from each nic-config template that is created by automated conversion tools such as convert_v1_net_data.py or convert_heat_nic_config_to_ansible_j2.py during the undercloud upgrade. If you do not remove these comment blocks, scaled-out nodes fail to provision, or you can lose control plane networking if network_config_update is set to true.

   {{# NOTE! Custom parameter {} was hard-coded in '
   'the converted template. To parameterize use the '
   '{{role.name}}ExtraGroupVars THT interface and update the '
   'template to use an ansible var. +#}}

4. On the main stack, copy the passwords.yaml file to the ~/overcloud-deploy/$(<stack>) directory. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-passwords.yaml ~/overcloud-deploy/<stack>/<stack>-passwords.yaml

5. On the main stack, copy the network-data.yaml file to the stack user’s home directory and deploy the networks. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-network-data.yaml ~/
   $ openstack overcloud network provision --debug \
   --output /home/stack/templates/generated-networks-deployed.yaml tripleo-<stack>-network-data.yaml

   For more information, see Provisioning and deploying your overcloud in Installing and managing Red Hat OpenStack Platform with director.

6. If your environment has pre-provisioned nodes, complete the following steps:

   1. Make a backup copy of the tripleo-<stack>-virtual-ips.yaml file in the stack user’s home directory:

      $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-virtual-ips.yaml ~/
      ~/overcloud-deploy/<stack>/tripleo-<stack>-virtual-ips.yaml.orig

   2. Include the following configuration at the end of the tripleo-<stack>-virtual-ips.yaml file:

      dns_name: overcloud
        ip_address: <ctrlplane_virtual_ip> 

      1

      
        name: control_virtual_ip 

      2

      
        network: ctlplane
        subnet: ctlplane-subnet

      1

      Replace <ctrlplane_virtual_ip> with the VIP of the control plane.

      2

      The default name of the VIP.

7. On the main stack, copy the virtual-ips.yaml file to the stack user’s home directory and provision the network VIPs. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-virtual-ips.yaml ~/
   $ openstack overcloud network vip provision --debug \
   --stack <stack> --output \
   /home/stack/templates/generated-vip-deployed.yaml tripleo-<stack>-virtual-ips.yaml

8. On the main stack, copy the network-environment.yaml file to the stack user’s home directory. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/<stack>-network-environment.yaml ~/templates/<stack>-network-environment.yaml

   Note

   Ensure that the NetworkConfigTemplate parameters point to the generated .j2 files and that you delete the resource_registry mappings in the overcloud-network-environment.yaml file. For more information, see Manually converting NIC templates to Jinja2 Ansible format in Customizing your Red Hat OpenStack Platform deployment.

9. On the main stack, copy the baremetal-deployment.yaml file to the stack user’s home directory and provision the overcloud nodes. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-baremetal-deployment.yaml ~/
   $ openstack overcloud node provision --debug --stack <stack> \
   --output /home/stack/templates/baremetal-deployment.yaml \
   tripleo-<stack>-baremetal-deployment.yaml

   Note

   This is the final step of the overcloud adoption. If your overcloud adoption takes longer than 10 minutes to complete, contact Red Hat Support.

10. Complete the following steps to prepare the containers:

    1. Back up the containers-prepare-parameter.yaml file that you used for the undercloud upgrade:

       $ cp containers-prepare-parameter.yaml \
       containers-prepare-parameter.yaml.orig

    2. Define the following environment variables before you run the script to update the containers-prepare-parameter.yaml file:

       • NAMESPACE: The namespace for the UBI9 images. For example, NAMESPACE='"namespace":"example.redhat.com:5002",'
       • EL8_NAMESPACE: The namespace for the UBI8 images.
       • NEUTRON_DRIVER: The driver to use and determine which OpenStack Networking (neutron) container to use. Set to the type of containers you used to deploy the original stack. For example, set to NEUTRON_DRIVER='"neutron_driver":"ovn",' to use OVN-based containers.

       • EL8_TAGS: The tags of the UBI8 images, for example, EL8_TAGS='"tag":"17.1",'.

         • Replace "17.1", with the tag that you use in your content view.

       • EL9_TAGS: The tags of the UBI9 images, for example, EL9_TAGS='"tag":"17.1",'.

         • Replace "17.1", with the tag that you use in your content view.

           For more information about the tag parameter, see Container image preparation parameters in Customizing your Red Hat OpenStack Platform deployment.

       • CONTROL_PLANE_ROLES: The list of control plane roles using the --role option, for example, --role ControllerOpenstack, --role Database, --role Messaging, --role Networker, --role CephStorage. To view the list of control plane roles in your environment, run the following command:

         $ export STACK=<stack> \
         $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
         /var/lib/mistral/${STACK}/tripleo-ansible-inventory.yaml \
         | grep -vi compute

         • Replace <stack> with the name of your stack.

       • COMPUTE_ROLES: The list of Compute roles using the --role option, for example, --Compute-1. To view the list of Compute roles in your environment, run the following command:

         $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
         /var/lib/mistral/${STACK}/tripleo-ansible-inventory.yaml \
         | grep -i compute

       • CEPH_OVERRIDE: If you deployed Red Hat Ceph Storage, specify the Red Hat Ceph Storage 5 container images. For example:

         CEPH_OVERRIDE='"ceph_image":"rhceph-5-rhel8","ceph_tag":"<latest>",'

         • Replace <latest> with the latest ceph_tag version, for example, 5-499.

           The following is an example of the containers-prepare-parameter.yaml file configuration:

           NAMESPACE='"namespace":"registry.redhat.io/rhosp-rhel9",'
           EL8_NAMESPACE='"namespace":"registry.redhat.io/rhosp-rhel8",'
           NEUTRON_DRIVER='"neutron_driver":"ovn",'
           EL8_TAGS='"tag":"17.1",'
           EL9_TAGS='"tag":"17.1",'
           CONTROL_PLANE_ROLES="--role Controller"
           COMPUTE_ROLES="--role Compute1 --role Compute2"
           CEPH_TAGS='"ceph_tag":"5",'

    3. Run the following script to to update the containers-prepare-parameter.yaml file:

       Warning

       If you deployed Red Hat Ceph Storage, ensure that the CEPH_OVERRIDE environment variable is set to the correct values before executing the following command. Failure to do so results in issues when upgrading Red Hat Ceph Storage.

       $ python3 /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \
            ${COMPUTE_ROLES} \
            ${CONTROL_PLANE_ROLES} \
            --enable-multi-rhel \
            --excludes collectd \
            --excludes nova-libvirt \
            --minor-override "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
            --major-override "{${EL9_TAGS}${NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
            --output-env-file \
           /home/stack/containers-prepare-parameter.yaml

       The multi-rhel-container-image-prepare.py script supports the following parameters:

       --output-env-file

       Writes the environment file that contains the default ContainerImagePrepare value.

       --local-push-destination

       Triggers an upload to a local registry.

       --enable-registry-login

       Enables the flag that allows the system to attempt to log in to a remote registry prior to pulling the containers. Use this flag when --local-push-destination is not used and the target systems have network connectivity to remote registries. Do not use this flag for an overcloud that might not have network connectivity to a remote registry.

       --enable-multi-rhel

       Enables multi-rhel.

       --excludes

       Lists the services to exclude.

       --major-override

       Lists the override parameters for a major release.

       --minor-override

       Lists the override parameters for a minor release.

       --role

       The list of roles.

       --role-file

       The role_data.yaml file.

    4. If you deployed Red Hat Ceph Storage, open the containers-prepare-parameter.yaml file to confirm that the Red Hat Ceph Storage 5 container images are specified and that there are no references to Red Hat Ceph Storage 6 container images.

11. If you have a director-deployed Red Hat Ceph Storage deployment, create a file called ceph_params.yaml and include the following content:

    parameter_defaults:
      CephSpecFqdn: true
      CephConfigPath: "/etc/ceph"
      CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms"
      DeployedCeph: true
      GrafanaPlugins: []

    Important

    Do not remove the ceph_params.yaml file after the RHOSP upgrade is complete. This file must be present in director-deployed Red Hat Ceph Storage environments. Additionally, any time you run openstack overcloud deploy, you must include the ceph_params.yaml file, for example, -e ceph_params.yaml.

    Note

    If your Red Hat Ceph Storage deployment includes short names, you must set the CephSpecFqdn parameter to false. If set to true, the inventory generates with both the short names and domain names, causing the Red Hat Ceph Storage upgrade to fail.

12. Create an environment file called upgrades-environment.yaml in your templates directory and include the following content:

    parameter_defaults:
      ExtraConfig:
        nova::workarounds::disable_compute_service_check_for_ffu: true
      DnsServers: ["<dns_servers>"]
      DockerInsecureRegistryAddress: [<undercloud_FQDN>]
      UpgradeInitCommand: |
        sudo subscription-manager repos --disable=*
          if $( grep -q  9.2  /etc/os-release )
          then
            sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-e4s-rpms --enable=rhel-9-for-x86_64-appstream-e4s-rpms --enable=rhel-9-for-x86_64-highavailability-e4s-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms
            sudo subscription-manager release --set=9.2
          else
            sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-aus-rpms --enable=rhel-8-for-x86_64-appstream-aus-rpms --enable=rhel-8-for-x86_64-highavailability-aus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
            sudo subscription-manager release --set=8.4
          fi
    
          if $(sudo podman ps | grep -q ceph )
          then
            sudo dnf -y install cephadm
          fi

    • Replace <dns_servers> with a comma-separated list of your DNS server IP addresses, for example, ["10.0.0.36", "10.0.0.37"].

    • Replace <undercloud_FQDN> with the fully qualified domain name (FQDN) of the undercloud host, for example, ["undercloud-0.ctlplane.redhat.local:8787"].

      For more information about the upgrade parameters that you can configure in the environment file, see Upgrade parameters.

13. On the undercloud, create a file called overcloud_upgrade_prepare.sh in your templates directory. You must create this file for each stack in your environment. This file includes the original content of your overcloud deploy file and the environment files that are relevant to your environment. For example:

    #!/bin/bash
    openstack overcloud upgrade prepare --yes \
      --timeout 460 \
      --templates /usr/share/openstack-tripleo-heat-templates \
      --ntp-server 192.168.24.1 \
      --stack <stack> \
      -r /home/stack/roles_data.yaml \
      -e /home/stack/templates/internal.yaml \
      -e /home/stack/templates/network/network-environment.yaml \
      -e /home/stack/templates/inject-trust-anchor.yaml \
      -e /home/stack/templates/hostnames.yml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
      -e /home/stack/templates/nodes_data.yaml \
      -e /home/stack/templates/debug.yaml \
      -e /home/stack/templates/firstboot.yaml \
      -e /home/stack/templates/upgrades-environment.yaml \
      -e /home/stack/overcloud-params.yaml \
      -e /home/stack/templates/overcloud-network-environment.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \
      -e /home/stack/skip_rhel_release.yaml \
      -e ~/containers-prepare-parameter.yaml \
      -e /home/stack/templates/baremetal-deployment.yaml \
      -e /home/stack/templates/generated-networks-deployed.yaml \
      -e /home/stack/templates/generated-vip-deployed.yaml

    1. In the original network-environment.yaml file (/home/stack/templates/network/network-environment.yaml), remove all the resource_registry resources that point to OS::TripleO::*::Net::SoftwareConfig.

    2. In the overcloud_upgrade_prepare.sh file, include the following options relevant to your environment:

       • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
       • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
       • The environment file (skip_rhel_release.yaml) with the release parameters (-e).
       • Any custom configuration environment files (-e) relevant to your deployment.
       • If applicable, your custom roles (roles_data) file by using --roles-file.
       • For Ceph deployments, the environment file (ceph_params.yaml) with the Ceph parameters (-e).
       • If applicable, the environment file (ipa-environment.yaml) with your IPA service (-e).
       • If you are using composable networks, the (network_data) file by using --network-file.
       • The files that were generated during overcloud adoption (networks-deployed.yaml, vip-deployed.yaml,baremetal-deployment.yaml) (-e). These files must be placed last in the overcloud upgrade prepare script.

       • If you use OVS, add the OVS environment files in the following order:

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml

       • If you use OVN, add the OVN environment files in the following order:

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dpdk.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml

           Note

           Do not include the network-isolation.yaml file in your overcloud deploy file or the overcloud_upgrade_prepare.sh file. Network isolation is defined in the network_data.yaml file.

       • If you use a custom stack name, pass the name with the --stack option.

         Note

         You must include the nova-hw-machine-type-upgrade.yaml file in your templates until all of your RHEL 8 Compute nodes are upgraded to RHEL 9 in the environment. If this file is excluded, an error appears in the nova_compute.log in the /var/log/containers/nova directory. After you upgrade all of your RHEL 8 Compute nodes to RHEL 9, you can remove this file from your configuration and update the stack.

    3. In the director-deployed Red Hat Ceph Storage use case, if you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must specify an additional environment file at the end of the overcloud_upgrade_prepare.sh script file. You must add the environment file at the end of the script because it overrides another environment file that is specified earlier in the script:

       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

    4. In the external Red Hat Ceph Storage use case, if you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must check that the associated environment file in the overcloud_upgrade_prepare.sh script points to the tripleo-based ceph-nfs role. If present, remove the following environment file:

       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

       And add the following environment file:

       -e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml

14. Run the upgrade preparation script for each stack in your environment:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    $ sh /home/stack/overcloud_upgrade_prepare.sh

15. Wait until the upgrade preparation completes.

16. Download the container images:

    $ openstack overcloud external-upgrade run --stack <stack> --tags container_image_prepare

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Verify that the following files that were exported during the undercloud upgrade contain the expected configuration for the overcloud upgrade. You can find the following files in the ~/overcloud-deploy/$(<stack>) directory:

   • tripleo-<stack>-network-data.yaml
   • tripleo-<stack>-virtual-ips.yaml

   • tripleo-<stack>-baremetal-deployment.yaml

     • Replace <stack> with the name of your stack.
     • Check that the mappings between the baremetal node UUIDs, hostnames, and IP addresses in the tripleo-<stack>-baremetal-deployment.yaml file match your current overcloud. Correct the mappings if required.
   Warning

   You must address the issues in Jinja2 comments, and then remove all Jinja2 comment blocks, from each nic-config template that is created by automated conversion tools such as convert_v1_net_data.py or convert_heat_nic_config_to_ansible_j2.py during the undercloud upgrade. If you do not remove these comment blocks, scaled-out nodes fail to provision, or you can lose control plane networking if network_config_update is set to true.

   {{# NOTE! Custom parameter {} was hard-coded in '
   'the converted template. To parameterize use the '
   '{{role.name}}ExtraGroupVars THT interface and update the '
   'template to use an ansible var. +#}}

4. On the main stack, copy the network-data.yaml file to the stack user’s home directory and deploy the networks. Repeat this step on each cell stack in your environment:

   Note

   The network-data.yaml file is available only on the overcloud stack. You must copy the file from the overcloud stack to all the other cell stacks.

   $ cp ~/overcloud-deploy/<overcloud-stack>/tripleo-<overcloud-stack>-network-data.yaml ~/
   $ openstack overcloud network provision --debug \
   --output /home/stack/templates/generated-<overcloud-stack>-networks-deployed.yaml tripleo-<overcloud-stack>-network-data.yaml

   For more information, see Provisioning and deploying your overcloud in Installing and managing Red Hat OpenStack Platform with director.

5. If your environment has pre-provisioned nodes, complete the following steps in each stack in your environment, starting with the overcloud stack:

   1. Make a backup copy of the tripleo-<overcloud-stack>-virtual-ips.yaml file in the stack user’s home directory:

      $ cp ~/overcloud-deploy/<overcloud-stack>/tripleo-<overcloud-stack>-virtual-ips.yaml ~/
      ~/overcloud-deploy/<overcloud-stack>/tripleo-<overcloud-stack>-virtual-ips.yaml.orig

   2. Include the following configuration at the end of the tripleo-<overcloud-stack>-virtual-ips.yaml file:

      dns_name: overcloud
        ip_address: <ctrlplane_virtual_ip> 

      1

      
        name: control_virtual_ip 

      2

      
        network: ctlplane
        subnet: ctlplane-subnet

      1

      Replace <ctrlplane_virtual_ip> with the VIP of the control plane.

      2

      The default name of the VIP.

6. On the overcloud stack, copy the virtual-ips.yaml file to the stack user’s home directory and provision the network VIPs. Repeat this step on each cell stack in your environment:

   $ cp ~/overcloud-deploy/<overcloud-stack>/tripleo-<overcloud-stack>-virtual-ips.yaml ~/
   $ openstack overcloud network vip provision --debug \
   --stack <overcloud-stack> --output \
   /home/stack/templates/generated-<overcloud-stack>-vip-deployed.yaml tripleo-<overcloud-stack>-virtual-ips.yaml

7. On the main stack, copy the baremetal-deployment.yaml file to the stack user’s home directory and provision the overcloud nodes. Repeat this step on each cell stack in your environment:

   $ cp ~/overcloud-deploy/<overcloud-stack>/tripleo-<overcloud-stack>-baremetal-deployment.yaml ~/
   $ openstack overcloud node provision --debug --stack <overcloud-stack> \
   --output /home/stack/templates/baremetal-<overcloud-stack>-deployment.yaml \
   tripleo-<overcloud-stack>-baremetal-deployment.yaml

   Note

   This is the final step of the overcloud adoption. If your overcloud adoption takes longer than 10 minutes to complete, contact Red Hat Support.

8. Complete the following steps to prepare the containers:

   1. Back up the containers-prepare-parameter.yaml file that you used for the undercloud upgrade:

      $ cp containers-prepare-parameter.yaml \
      containers-prepare-parameter.yaml.orig

   2. Define the following environment variables before you run the script to update the containers-prepare-parameter.yaml file:

      • NAMESPACE: The namespace for the UBI9 images. For example, NAMESPACE='"namespace":"example.redhat.com:5002",'
      • EL8_NAMESPACE: The namespace for the UBI8 images.
      • NEUTRON_DRIVER: The driver to use and determine which OpenStack Networking (neutron) container to use. Set to the type of containers you used to deploy the original stack. For example, set to NEUTRON_DRIVER='"neutron_driver":"ovn",' to use OVN-based containers.

      • EL8_TAGS: The tags of the UBI8 images, for example, EL8_TAGS='"tag":"17.1",'.

        • Replace "17.1", with the tag that you use in your content view.

      • EL9_TAGS: The tags of the UBI9 images, for example, EL9_TAGS='"tag":"17.1",'.

        • Replace "17.1", with the tag that you use in your content view.

          For more information about the tag parameter, see Container image preparation parameters in Customizing your Red Hat OpenStack Platform deployment.

      • CONTROL_PLANE_ROLES: The list of control plane roles using the --role option, for example, --role ControllerOpenstack, --role Database, --role Messaging, --role Networker, --role CephStorage. To view the list of control plane roles in your environment, run the following command:

        $ export STACK=<stack> \
        $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
        /var/lib/mistral/${STACK}/tripleo-ansible-inventory.yaml \
        | grep -vi compute

        • Replace <stack> with the name of your stack.

      • COMPUTE_ROLES: The list of Compute roles using the --role option, for example, --Compute-1. To view the list of Compute roles in your environment, run the following command:

        $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
        /var/lib/mistral/${STACK}/tripleo-ansible-inventory.yaml \
        | grep -i compute

      • CEPH_OVERRIDE: If you deployed Red Hat Ceph Storage, specify the Red Hat Ceph Storage 5 container images. For example:

        CEPH_OVERRIDE='"ceph_image":"rhceph-5-rhel8","ceph_tag":"<latest>",'

        • Replace <latest> with the latest ceph_tag version, for example, 5-499.

          The following is an example of the containers-prepare-parameter.yaml file configuration:

          NAMESPACE='"namespace":"registry.redhat.io/rhosp-rhel9",'
          EL8_NAMESPACE='"namespace":"registry.redhat.io/rhosp-rhel8",'
          NEUTRON_DRIVER='"neutron_driver":"ovn",'
          EL8_TAGS='"tag":"17.1",'
          EL9_TAGS='"tag":"17.1",'
          CONTROL_PLANE_ROLES="--role Controller"
          COMPUTE_ROLES="--role Compute1 --role Compute2"
          CEPH_TAGS='"ceph_tag":"5",'

   3. Run the following script to to update the containers-prepare-parameter.yaml file:

      Warning

      If you deployed Red Hat Ceph Storage, ensure that the CEPH_OVERRIDE environment variable is set to the correct values before executing the following command. Failure to do so results in issues when upgrading Red Hat Ceph Storage.

      $ python3 /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \
           ${COMPUTE_ROLES} \
           ${CONTROL_PLANE_ROLES} \
           --enable-multi-rhel \
           --excludes collectd \
           --excludes nova-libvirt \
           --minor-override "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
           --major-override "{${EL9_TAGS}${NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
           --output-env-file \
          /home/stack/containers-prepare-parameter.yaml

      The multi-rhel-container-image-prepare.py script supports the following parameters:

      --output-env-file

      Writes the environment file that contains the default ContainerImagePrepare value.

      --local-push-destination

      Triggers an upload to a local registry.

      --enable-registry-login

      Enables the flag that allows the system to attempt to log in to a remote registry prior to pulling the containers. Use this flag when --local-push-destination is not used and the target systems have network connectivity to remote registries. Do not use this flag for an overcloud that might not have network connectivity to a remote registry.

      --enable-multi-rhel

      Enables multi-rhel.

      --excludes

      Lists the services to exclude.

      --major-override

      Lists the override parameters for a major release.

      --minor-override

      Lists the override parameters for a minor release.

      --role

      The list of roles.

      --role-file

      The role_data.yaml file.

   4. If you deployed Red Hat Ceph Storage, open the containers-prepare-parameter.yaml file to confirm that the Red Hat Ceph Storage 5 container images are specified and that there are no references to Red Hat Ceph Storage 6 container images.

9. If you have a director-deployed Red Hat Ceph Storage deployment, create a file called ceph_params.yaml and include the following content:

   parameter_defaults:
     CephSpecFqdn: true
     CephConfigPath: "/etc/ceph"
     CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms"
     DeployedCeph: true
     GrafanaPlugins: []

   Important

   Do not remove the ceph_params.yaml file after the RHOSP upgrade is complete. This file must be present in director-deployed Red Hat Ceph Storage environments. Additionally, any time you run openstack overcloud deploy, you must include the ceph_params.yaml file, for example, -e ceph_params.yaml.

   Note

   If your Red Hat Ceph Storage deployment includes short names, you must set the CephSpecFqdn parameter to false. If set to true, the inventory generates with both the short names and domain names, causing the Red Hat Ceph Storage upgrade to fail.

10. Create an environment file called upgrades-environment.yaml in your templates directory and include the following content:

    parameter_defaults:
      ExtraConfig:
        nova::workarounds::disable_compute_service_check_for_ffu: true
      DnsServers: ["<dns_servers>"]
      DockerInsecureRegistryAddress: [<undercloud_FQDN>]
      UpgradeInitCommand: |
        sudo subscription-manager repos --disable=*
          if $( grep -q  9.2  /etc/os-release )
          then
            sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-e4s-rpms --enable=rhel-9-for-x86_64-appstream-e4s-rpms --enable=rhel-9-for-x86_64-highavailability-e4s-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms
            sudo subscription-manager release --set=9.2
          else
            sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-aus-rpms --enable=rhel-8-for-x86_64-appstream-aus-rpms --enable=rhel-8-for-x86_64-highavailability-aus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
            sudo subscription-manager release --set=8.4
          fi
    
          if $(sudo podman ps | grep -q ceph )
          then
            sudo dnf -y install cephadm
          fi

    • Replace <dns_servers> with a comma-separated list of your DNS server IP addresses, for example, ["10.0.0.36", "10.0.0.37"].

    • Replace <undercloud_FQDN> with the fully qualified domain name (FQDN) of the undercloud host, for example, ["undercloud-0.ctlplane.redhat.local:8787"].

      For more information about the upgrade parameters that you can configure in the environment file, see Upgrade parameters.

11. On the undercloud, create a file called overcloud_upgrade_prepare.sh in your templates directory. You must create this file for each stack in your environment, starting with the overcloud stack. This file includes the original content of your overcloud deploy file and the environment files that are relevant to each cell stack. For example:

    #!/bin/bash
    openstack overcloud upgrade prepare --yes \
      --timeout 460 \
      --templates /usr/share/openstack-tripleo-heat-templates \
      --ntp-server 192.168.24.1 \
      --stack <overcloud-stack> \
      -r /home/stack/roles_data.yaml \
      -e /home/stack/templates/internal.yaml \
      -e /home/stack/templates/network/network-environment.yaml \
      -e /home/stack/templates/inject-trust-anchor.yaml \
      -e /home/stack/templates/hostnames.yml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
      -e /home/stack/templates/nodes_data.yaml \
      -e /home/stack/templates/debug.yaml \
      -e /home/stack/templates/firstboot.yaml \
      -e /home/stack/templates/upgrades-environment.yaml \
      -e /home/stack/overcloud-params.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \
      -e /home/stack/skip_rhel_release.yaml \
      -e ~/containers-prepare-parameter.yaml \
      -e /home/stack/templates/baremetal-<overcloud-stack>-deployment.yaml \
      -e /home/stack/templates/generated-<overcloud-stack>-networks-deployed.yaml \
      -e /home/stack/templates/generated-<overcloud-stack>-vip-deployed.yaml

    1. In the original network-environment.yaml file (/home/stack/templates/network/network-environment.yaml), remove all the resource_registry resources that point to OS::TripleO::*::Net::SoftwareConfig.

    2. In the overcloud_upgrade_prepare.sh file, include the following options relevant to your environment:

       • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
       • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
       • The default_cell_export.yaml file that you exported from the default cell in the overcloud stack. You must add this file to all of the overcloud_upgrade_prepare.sh files for each cell stack, except for the overcloud stack.
       • The environment file (skip_rhel_release.yaml) with the release parameters (-e).
       • Any custom configuration environment files (-e) relevant to your deployment.
       • If applicable, your custom roles (roles_data) file by using --roles-file.
       • For Ceph deployments, the environment file (ceph_params.yaml) with the Ceph parameters (-e).
       • If applicable, the environment file (ipa-environment.yaml) with your IPA service (-e).
       • If you are using composable networks, the (network_data) file by using --network-file.
       • The files that were generated during overcloud adoption (networks-deployed.yaml, vip-deployed.yaml,baremetal-deployment.yaml) (-e). These files must be placed last in the overcloud upgrade prepare script.

       • If you use OVS, add the OVS environment files in the following order:

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml

       • If you use OVN, add the OVN environment files in the following order:

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dpdk.yaml
         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml

         • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml

           Note

           Do not include the network-isolation.yaml file in your overcloud deploy file or the overcloud_upgrade_prepare.sh file. Network isolation is defined in the network_data.yaml file.

       • If you use a custom stack name, pass the name with the --stack option.

         Note

         You must include the nova-hw-machine-type-upgrade.yaml file in your templates until all of your RHEL 8 Compute nodes are upgraded to RHEL 9 in the environment. If this file is excluded, an error appears in the nova_compute.log in the /var/log/containers/nova directory. After you upgrade all of your RHEL 8 Compute nodes to RHEL 9, you can remove this file from your configuration and update the stack.

    3. In the director-deployed Red Hat Ceph Storage use case, if you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must specify an additional environment file at the end of the overcloud_upgrade_prepare.sh script file. You must add the environment file at the end of the script because it overrides another environment file that is specified earlier in the script:

       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

    4. In the external Red Hat Ceph Storage use case, if you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must check that the associated environment file in the overcloud_upgrade_prepare.sh script points to the tripleo-based ceph-nfs role. If present, remove the following environment file:

       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

       And add the following environment file:

       -e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml

12. Run the upgrade preparation script on the overcloud stack:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    $ sh /home/stack/overcloud_upgrade_prepare.sh

    Wait until the upgrade preparation completes.

13. Export the cell configuration and password information from the default cell in the overcloud stack to a new common environment file for the multi-cell deployment. You must include the /home/stack/common/default_cell_export.yaml file and all of the environment files that are relevant to your environment in the script for each cell:

    (undercloud) $ openstack overcloud cell export --control-plane-stack overcloud \
     -f --output-file common/default_cell_export.yaml \
     --working-dir /home/stack/overcloud-deploy/overcloud/

14. If you have a DCN environment, the generated cell_export.yaml file is incomplete. To get the correct file, complete the following steps:

    1. Source the stackrc undercloud credentials file:

       $ source ~/stackrc

    2. Force the necessary data to be generated into the stack’s overcloud-deploy directory:

       $ openstack overcloud upgrade run --yes --limit allovercloud,undercloud --stack overcloud --playbook overcloud/upgrade_steps_playbook.yaml --skip-tags upgrade_steps

    3. Re-export the overcloud stack to to get the updated cell_export.yaml file to use in the overcloud upgrade prepare script:

       $ openstack overcloud cell export --control-plane-stack overcloud --force-overwrite --working-dir overcloud-deploy/overcloud --output-file overcloud-deploy/overcloud/overcloud-cell-export.yaml

15. Run the upgrade preparation script on each cell stack. For example:

    $ source stackrc
    $ chmod 755 /home/stack/<stack1>-overcloud_upgrade_prepare.sh
    $ sh /home/stack/<stack1>-overcloud_upgrade_prepare.sh

    Wait until the upgrade preparation completes.

16. Download the container images for each cell stack, starting with the overcloud stack:

    $ openstack overcloud external-upgrade run --stack <stack> --tags container_image_prepare

Procedure

1. Enable the Ceph 5 Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to the undercloud node as the stack user.

2. Download the Red Hat Ceph Storage container image from the Satellite Server:

   $ sudo podman pull <ceph_image_file>

   • Replace <ceph_image_file> with the Red Hat Ceph Storage container image file hosted on the Satellite Server. The following is an example of this command:

     $ sudo podman pull satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   • Replace <stack> with the name of your stack.
   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

4. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific.

   $ sudo podman exec ceph-mon-$(hostname -f) ceph versions
   {
       "mon": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "mgr": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "osd": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180
       },
       "mds": {},
       "rgw": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3
       },
       "overall": {
           "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189
       }
   }

   Note

   The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container.

5. Create the ceph-admin user and distribute the appropriate keyrings:

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd

6. Update the packages on the Red Hat Ceph Storage nodes:

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_ansible_remote_tmp \
       --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default).

7. Configure the Red Hat Ceph Storage nodes to use cephadm:

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

   • If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter.

     Note

     The adoption of cephadm can cause downtime in the RGW and Alertmanager services. For more information about these issues, see Restarting Red Hat Ceph Storage 5 services.

8. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node.

   Important

   The command in the previous step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure.

   The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm. This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm.

   $ sudo cephadm shell -- ceph -s
     cluster:
       id:     f5a40da5-6d88-4315-9bb3-6b16df51d765
       health: HEALTH_WARN
               63 stray daemon(s) not managed by cephadm

9. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file:

   Important

   Do not include ceph-ansible environment or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml, in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy. For more information about replacing ceph-ansible environment and deployment files with cephadm files, see Implications of upgrading to Red Hat Ceph Storage 5.

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e <cephadm-file> \
     -e ~/containers-prepare-parameter.yaml

   where:

   <cephadm-file>

   • If you deployed RGW in a previous RHOSP version, or if you plan to deploy RGW, use environments/cephadm/cephadm.yaml.
   • If you plan to deploy RBD, use environments/cephadm/cephadm-rbd-only.yaml.

10. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

11. Save the file.

12. Run the upgrade preparation command:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

13. If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node:

    1. Log in to a Controller node:

       $ ssh cloud-admin@<controller_ip>

       • Replace <controller_ip> with the IP address of the Controller node.

    2. Retrieve a cephadm shell from the Controller node:

       Example

       [cloud-admin@controller-0 ~]$ sudo cephadm shell

    3. In the cephadm shell, create a temporary hci.conf file:

       Example

       [ceph: root@edpm-controller-0 /]# cat <<EOF > hci.conf
       [osd]
       osd_memory_target_autotune = true
       osd_numa_auto_affinity = true
       [mgr]
       mgr/cephadm/autotune_memory_target_ratio = 0.2
       EOF
       …​

    4. Apply the configuration:

       Example

       [ceph: root@edpm-controller-0 /]# ceph config assimilate-conf -i hci.conf

       For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the overcloud upgrade. Choose one of the following upgrade methods based on your specific environment:

   • If you do not have a DCN or multi-cell deployment that includes ML2/OVN, upgrade RHOSP on all nodes in your main stack:

     1. Upgrade RHOSP on all nodes in your main stack:

        $ openstack overcloud upgrade run --yes --stack <stack> --debug --limit allovercloud,undercloud --playbook all

        Important

        Do not modify the --limit option. You must upgrade all nodes in the stack at once to avoid workload disruption. For more information about the importance of upgrading all of your overcloud nodes at the same time, see The importance of upgrading Red Hat OpenStack Platform on all overcloud nodes at once.

        • Replace <stack> with the name of the overcloud stack that you want to upgrade the nodes on.

          Repeat this step for each stack in your RHOSP deployment.

   • If you have a DCN or multi-cell deployment that includes ML2/OVN, complete the following steps:

     1. Upgrade the OVN containers and all host packages on each stack. For both DCN and multi-cell deployments, you must upgrade the central stack and control stack first, respectively:

        $ openstack overcloud upgrade run --stack <stack> --tags setup_packages,ovn --limit allovercloud --yes

        • Replace <stack> with the name of the stack you are upgrading.

          Important

          After you run the RHOSP upgrade on the central or control stack, the OVN DBs are migrated from a Pacemaker deployment to an Ansible-controlled cluster. As a result, the endpoint that the OVN controllers connect to changes. This connection is updated during the RHOSP upgrade on each additional stack. Depending on the size of your environment, some stacks might be updated later, which can result in a data plane outage. To avoid this issue, update the connection to use the previous endpoint and new endpoint at the same time. The following example shows the workaround for a multi-cell environment. The workaround is the same for DCN environments that include OVN. Ensure that you export the overcloud-export.yaml file first:

          $ cat <<'EOF' > ~/ovn_workaround.yaml
          # Playbook
          - become: true
            hosts: '{{ ovn_compute_role }}'
            strategy: tripleo_free
            name: OVN workaround playbook
            tasks:
             - name: Read ovn southbound port
               command: puppet lookup --facts /etc/puppet/hieradata/service_configs.json ovn::southbound::port --render-as s
               register: ovn_southbound_port
               failed_when: false
             - name: Read ovn vip
               command: puppet lookup --facts /etc/puppet/hieradata/all_nodes.json ovn_dbs_vip --render-as s
               register: ovn_dbs_vip
             - name: Create the new connection
               set_fact:
                  new_ovn_connection: "ssl:{{ovn_dbs_vip.stdout}}:{{ovn_southbound_port.stdout}},ssl:{{parameter_defaults.AllNodesExtraMapData.ovn_dbs_node_ips[0]}}:{{ovn_southbound_port.stdout}},ssl:{{parameter_defaults.AllNodesExtraMapData.ovn_dbs_node_ips[1]}}:{{ovn_southbound_port.stdout}},ssl:{{parameter_defaults.AllNodesExtraMapData.ovn_dbs_node_ips[2]}}:{{ovn_southbound_port.stdout}}"
             - name: showing the new connection
               debug:
                 msg: "the new connection is {{ new_ovn_connection }}"
               tags:
                 - debug_msg
             - name: Get ovn connection
               command: ovs-vsctl get Open_vSwitch . external_ids:ovn-remote
               register: ovn_connection
             - set_fact:
                 change_needed: "{{ ovn_connection.stdout.split(',') | length == 1 }}"
             - name: Reconfigure ovn connection
               when: change_needed|bool
               command: ovs-vsctl set Open_vSwitch . external_ids:ovn-remote="{{new_ovn_connection}}"
               tags:
                 - apply_conf
             - name: Reconfigure ovn connection for metadata agents on computes
               when: change_needed|bool
               shell: |
                 crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini  ovn ovn_sb_connection "{{ new_ovn_connection }}"
                 systemctl restart tripleo_ovn_metadata_agent
               tags:
                 - apply_conf
          EOF

          $ ansible-playbook -i overcloud-deploy/$CELLSTACK/tripleo-ansible-inventory.yaml -e @~/overcloud-deploy/overcloud/overcloud-export.yaml -e ovn_compute_role=Compute ovn_workaround.yaml

     2. Verify that the ovn-controller is updated on all overcloud nodes:

        $ sudo podman ps | grep ovn_controller

        Sample output

        5ddc21ef9056  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-ovn-controller:17.1_20230905.1
        kolla_start  20 hours ago  Up 20 hours (healthy)  ovn_controller

     3. Upgrade the service containers and update the host packages on each stack, starting with the central stack or control stack:

        $ openstack overcloud upgrade run --stack <stack> --skip-tags ovn --limit allovercloud --yes

        Note

        To avoid a long maintenance window, you can run the upgrade only on the CellController role in your cell stack first, and then run the upgrade on all the nodes in the cell stack in the next maintenance window. For example:

        First maintenance window:

        $ openstack overcloud upgrade run --stack AZ2 --skip-tags ovn --limit CellController --yes

        Second maintenance window:

        $ openstack overcloud upgrade run --stack AZ2 --skip-tags ovn --limit allovercloud --yes

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

   $ sudo grep PermitRootLogin /etc/ssh/sshd_config

3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

   PermitRootLogin no

4. Save the file.

Procedure

1. Validate your SSH key size:

   ssh-keygen -l -f /home/stack/overcloud-deploy/overcloud/ssh_private_key

   Example output:

   1024 SHA256:Xqz0Xz0/aJua6B3qRD7VsLr6n/V3zhmnGSkcFR6FlJw stack@director.example.local (RSA)

2. If your SSH key is less than 2048 bits, you must rotate out the SSH key before continuing. For more information, see Updating SSH keys in your OpenStack environment in Hardening Red Hat OpenStack Platform.

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file named system_upgrade.yaml in your templates directory and include the following content:

   parameter_defaults:
     UpgradeLeappDevelSkip: "LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_SKIP_CHECK_OS_RELEASE=1 LEAPP_NO_NETWORK_RENAMING=1 LEAPP_DEVEL_TARGET_RELEASE=9.2 LEAPP_TARGET_PRODUCT_CHANNEL=e4s"
     UpgradeLeappDebug: false
     UpgradeLeappEnabled: true
     LeappActorsToRemove: ['checkifcfg','persistentnetnamesdisable','checkinstalledkernels','biosdevname']
     LeappRepoInitCommand: |
        subscription-manager repos --disable=*
        subscription-manager repos --enable rhel-8-for-x86_64-baseos-aus-rpms --enable rhel-8-for-x86_64-appstream-aus-rpms --enable openstack-17.1-for-rhel-8-x86_64-rpms
        subscription-manager release --set=8.4
     UpgradeLeappCommandOptions: "--enablerepo=rhel-9-for-x86_64-baseos-e4s-rpms --enablerepo=rhel-9-for-x86_64-appstream-e4s-rpms --enablerepo=rhel-9-for-x86_64-highavailability-e4s-rpms --enablerepo=openstack-17.1-for-rhel-9-x86_64-rpms --enablerepo=fast-datapath-for-rhel-9-x86_64-rpms"

   Note

   If your deployment includes Red Hat Ceph Storage nodes, you must add the CephLeappRepoInitCommand parameter and specify the source OS version of your Red Hat Ceph Storage nodes. For example:

   CephLeappRepoInitCommand:
   ...
     subscription-manager release --set=8.6

3. Add the LeappInitCommand parameter to your system_upgrade.yaml file to specify additional requirements applicable to your environment, for example, if you need to define role-based overrides:

     LeappInitCommand: |
       subscription-manager repos --disable=*
       subscription-manager release --unset
       subscription-manager repos --enable=rhel-9-for-x86_64-baseos-e4s-rpms --enable=rhel-9-for-x86_64-appstream-e4s-rpms --enable=rhel-9-for-x86_64-highavailability-e4s-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
       leapp answer --add --section check_vdo.confirm=True
   
       dnf -y remove ruby-irb

   Important

   Removing the ruby-irb package is mandatory to avoid a conflict between the RHEL 8 ruby-irb directory and the RHEL 9 symlink. For more information, see the Red Hat Knowledgebase solution leapp upgrade RHEL8 to RHEL9 fails with error "rubygem-irb-1.3.5-160.el9_0.noarch conflicts with file from package ruby-irb-2.5.9-110.module+el8.6.0+15956+aa803fc1.noarch".

   Note

   If your environment previously ran RHOSP 13.0 or earlier, during the system upgrade, a known issue causes GRUB to contain RHEL 7 entries instead of RHEL 8 entries. For more information, including a workaround, see the Red Hat Knowledgebase solution Openstack 16 to 17 FFU - During LEAPP upgrade UEFI systems do not boot due to invalid /boot/grub2/grub.cfg.

4. If you are upgrading Red Hat Ceph Storage nodes that originated in RHOSP 16.1 and earlier, add the CephStorageLeappInitCommand parameter to remove fencing agents:

     LeappInitCommand: |
       ...
   
       dnf -y remove ruby-irb
   
     CephStorageLeappInitCommand: |
       ...
   
       dnf -y remove ruby-irb fence-agents-common

5. If you use kernel-based NIC names, add the following parameter to the system_upgrade.yaml file to ensure that the NIC names persist throughout the upgrade process:

   parameter_defaults:
     NICsPrefixesToUdev: ['en']
   ...

6. Run the Leapp upgrade:

   $ openstack undercloud upgrade --yes --system-upgrade \
   /home/stack/system_upgrade.yaml

   Note

   If you need to run the Leapp upgrade again, you must first reset the repositories to RHEL 8.

7. Reboot the undercloud:

   $ sudo reboot

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the following script without the CONTROL_PLANE_ROLES parameter. Ensure that you include the variables that you used to prepare the containers in Running the overcloud upgrade preparation.

   python3 \
   /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \
        ${COMPUTE_ROLES} \
        --enable-multi-rhel \
        --excludes collectd \
        --excludes nova-libvirt \
        --minor-override \
   "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
        --major-override \
        "{${EL9_TAGS}${NAMESPACE}${CEPH_OVERRIDE}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
        --output-env-file \
   /home/stack/containers-prepare-parameter.yaml

   Note

   The CONTROL_PLANE_ROLES parameter defines the list of your control plane roles. Removing this parameter from the script prepares the control plane roles for an upgrade to RHEL 9.2. If the CONTROL_PLANE_ROLES parameter is included in the script, the control plane roles remain on RHEL 8.4.

4. In the skip_rhel_release.yaml file, set the SkipRhelEnforcement parameter to false:

   parameter_defaults:
     SkipRhelEnforcement: false

5. Update the overcloud_upgrade_prepare.sh file:

   $ openstack overcloud upgrade prepare --yes \
       ...
       -e /home/stack/system_upgrade.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /home/stack/skip_rhel_release.yaml \
       ...

   • Include the system_upgrade.yaml file with the upgrade-specific parameters (-e).
   • Include the containers-prepare-parameter.yaml file with the control plane roles removed (-e).
   • Include the skip_rhel_release.yaml file with the release parameters (-e).

6. Run the overcloud_upgrade_prepare.sh script:

   $ sh /home/stack/overcloud_upgrade_prepare.sh

7. Fetch any new or modified containers that you require for the system upgrade:

   $ openstack overcloud external-upgrade run  \
        --stack <stack> \
        --tags container_image_prepare 2>&1

8. Upgrade the first one-third of the control plane nodes:

   $ openstack overcloud upgrade run --yes \
        --stack <stack> \
        --tags system_upgrade \
        --limit <controller-0>,<database-0>,<messaging-0>,<networker-0>,<ceph-0>

   • Replace <stack> with the name of your stack.
   • Replace <controller-0>,<database-0>,<messaging-0>,<networker-0>,<ceph-0> with your own node names.

9. Log in to each upgraded node and verify that the cluster in each node is running:

   $ sudo pcs status

   Repeat this verification step after you upgrade the second one-third of your control plane nodes, and after you upgrade the last one-third of your control plane nodes.

10. Upgrade the second one-third of the control plane nodes:

    $ openstack overcloud upgrade run --yes \
         --stack <stack> \
         --tags system_upgrade \
         --limit <controller-1>,<database-1>,<messaging-1>,<networker-1>,<ceph-1>

    • Replace <controller-1>,<database-1>,<messaging-1>,<networker-1>,<ceph-1> with your own node names.

11. Upgrade the last one-third of the control plane nodes:

    $ openstack overcloud upgrade run --yes \
         --stack <stack> \
         --tags system_upgrade \
         --limit <controller-2>,<database-2>,<messaging-2>,<networker-2>,<ceph-2>

    • Replace <controller-2>,<database-2>,<messaging-2>,<networker-2>,<ceph-2> with your own node names.

12. If you enabled STF, you must update the collectd container on all nodes after the operating system upgrade:

    1. Run the upgrade command with no tags:

       $ openstack overcloud upgrade run --yes \
            --stack <stack> \
            --limit <undercloud>,<controller-0>,<controller-1>,<controller-2>,<database-0>,<database-1>,<database-2>,<networker-0>,<networker-1>,<networker-2>,<ceph-0>,<ceph-1>,<ceph-2>

       • Replace <undercloud>,<controller-0>,<controller-1>,<controller-2>,<database-0>,<database-1>,<database-2>,<networker-0>,<networker-1>,<networker-2>,<ceph-0>,<ceph-1>,<ceph-2> with your own node names.

    2. When the upgrade process is complete, check the status of the collectd container on each overcloud node:

       $ sudo podman ps | grep collectd

    3. If the collectd container is not running on the overcloud nodes, then manually start the collectd container on all the overcloud nodes:

       $ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -m shell -a "sudo podman restart collectd

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In the container-image-prepare.yaml file, ensure that only the tags specified in the ContainerImagePrepare parameter are included, and the MultiRhelRoleContainerImagePrepare parameter is removed. For example:

   parameter_defaults:
     ContainerImagePrepare:
     - tag_from_label: "{version}-{release}"
       set:
         namespace:
         name_prefix:
         name_suffix:
         tag:
         rhel_containers: false
         neutron_driver: ovn
         ceph_namespace:
         ceph_image:
         ceph_tag:

4. In the roles_data.yaml file, replace the OS::TripleO::Services::NovaLibvirtLegacy service with the OS::TripleO::Services::NovaLibvirt service that is required for RHEL 9.2.

5. Include the -e system_upgrade.yaml argument and the other required -e environment file arguments in the overcloud_upgrade_prepare.sh script as shown in the following example:

   $ openstack overcloud upgrade prepare --yes
   …​
   -e /home/stack/system_upgrade.yaml
   …​

6. Run the overcloud_upgrade_prepare.sh script.

7. Upgrade the operating system on the Compute nodes to RHEL 9.2. Use the --limit option with a comma-separated list of nodes that you want to upgrade. The following example upgrades the compute-0, compute-1, and compute-2 nodes.

   $ openstack overcloud upgrade run --yes --tags system_upgrade --stack <stack> --limit compute-0,compute-1,compute-2

   • Replace <stack> with the name of your stack.

8. Upgrade the containers on the Compute nodes to RHEL 9.2. Use the --limit option with a comma-separated list of nodes that you want to upgrade. The following example upgrades the compute-0, compute-1, and compute-2 nodes.

   $ openstack overcloud upgrade run --yes --stack <stack>  --limit compute-0,compute-1,compute-2

1. Plan which nodes you want to upgrade to RHEL 9.2, and which nodes you want to remain on RHEL 8.4. Choose a role name for each role that you are creating for each batch of nodes, for example, ComputeRHEL-9.2 and ComputeRHEL-8.4.

2. Create roles that store the nodes that you want to upgrade to RHEL 9.2, or the nodes that you want to stay on RHEL 8.4. These roles can remain empty until you are ready to move your Compute nodes to a new role. You can create as many roles as you need and divide nodes among them any way you decide. For example:

   • If your environment uses a role called ComputeSRIOV and you need to run a canary test to upgrade to RHEL 9.2, you can create a new ComputeSRIOVRHEL9 role and move the canary node to the new role.
   • If your environment uses a role called ComputeOffload and you want to upgrade most nodes in that role to RHEL 9.2, but keep a few nodes on RHEL 8.4, you can create a new ComputeOffloadRHEL8 role to store the RHEL 8.4 nodes. You can then select the nodes in the original ComputeOffload role to upgrade to RHEL 9.2.
3. Move the nodes from each Compute role to the new role.

4. Upgrade the operating system on specific Compute nodes to RHEL 9.2. You can upgrade nodes in batches from the same role or multiple roles.

   Note

   In a Multi-RHEL environment, the deployment should continue to use the pc-i440fx machine type. Do not update the default to Q35. Migrating to the Q35 machine type is a separate, post-upgrade procedure to follow after all Compute nodes are upgraded to RHEL 9.2. For more information about migrating the Q35 machine type, see Updating the default machine type for hosts after an upgrade to RHOSP 17.

   Use the following procedures to upgrade Compute nodes to a Multi-RHEL environment:

   • Creating roles for Multi-RHEL Compute nodes
   • Upgrading the Compute node operating system

Procedure

1. Check the list of RPMs that you installed and ensure that there are no RHOSP 16.2 or legacy 17.1 images, for example, rhosp-director-images-x86_64-17.1:

   $ rpm -qa | egrep "rhosp-director-images-*-16.2|rhosp-director-images-x86_64-17.1"

   If the list displays these images, remove them by running the following command:

   $ sudo dnf -y remove rhosp-director-images-*-16.2 rhosp-director-images-x86_64-17.1

2. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-hardened-uefi-full.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-17.1.tar; do tar -xvf $i; done
   $ cd ~

4. Import the images into director:

   (undercloud) [stack@director images]$ openstack overcloud image upload --image-path /home/stack/images/ --update-existing

   The command completes the following tasks:

   • Converts the image format from QCOW to RAW.
   • Provides status updates about the upload of the image.

Procedure

1. Log in to the undercloud as the stack user.

2. If your Compute nodes support simultaneous multithreading (SMT) but you created instances with the hw:cpu_thread_policy=isolated policy, you must perform one of the following options:

   • Create a new flavor that does not set the hw:cpu_thread_policy thread policy and resize the instances with that flavor:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Create a flavor with the default thread policy, prefer:

        (overcloud) $ openstack flavor create <flavor>

        Note

        When you resize an instance, you must use a new flavor. You cannot reuse the current flavor. For more information, see Resizing an instance in the Creating and managing instances guide.

     3. Convert the instances to use the new flavor:

        (overcloud) $ openstack server resize --flavor <flavor> <server>
        (overcloud) $ openstack server resize confirm <server>

     4. Repeat this step for all pinned instances that use the hw:cpu_thread_policy=isolated policy.

   • Migrate instances from the Compute node and disable SMT on the Compute node:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Disable the Compute node from accepting new virtual machines:

        (overcloud) $ openstack compute service list
        (overcloud) $ openstack compute service set <hostname> nova-compute --disable

     3. Migrate all instances from the Compute node. For more information on instance migration, see Migrating virtual machine instances between Compute nodes.
     4. Reboot the Compute node and disable SMT in the BIOS of the Compute node.
     5. Boot the Compute node.

     6. Re-enable the Compute node:

        (overcloud) $ openstack compute service set <hostname> nova-compute --enable

3. Source the stackrc file:

   $ source ~/stackrc

4. Edit the environment file that contains the NovaVcpuPinSet parameter.

5. Migrate the CPU pinning configuration from the NovaVcpuPinSet parameter to NovaComputeCpuDedicatedSet and NovaComputeCpuSharedSet:

   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuDedicatedSet for hosts that were previously used for pinned instances.
   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuSharedSet for hosts that were previously used for unpinned instances.
   • If there is no value set for NovaVcpuPinSet, then all Compute node cores should be assigned to either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, depending on the type of instances you intend to host on the nodes.

   For example, your previous environment file might contain the following pinning configuration:

   parameter_defaults:
     ...
     NovaVcpuPinSet: 1,2,3,5,6,7
     ...

   To migrate the configuration to a pinned configuration, set the NovaComputeCpuDedicatedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuDedicatedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   To migrate the configuration to an unpinned configuration, set the NovaComputeCpuSharedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuSharedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   Important

   Ensure the configuration of either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet matches the configuration defined in NovaVcpuPinSet. To change the configuration for either of these, or to configure both NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, ensure the Compute nodes with the pinning configuration are not running any instances before updating the configuration.

6. Save the file.

7. Run the deployment command to update the overcloud with the new CPU pinning parameters.

   (undercloud) $ openstack overcloud deploy \
       --stack _STACK NAME_ \
       --templates \
       ...
       -e /home/stack/templates/<compute_environment_file>.yaml
       ...

Procedure

1. Identify the instances that have no machine type recorded in the database:

   $nova-manage libvirt list_unset_machine_type

   1. If you find instances that have no machine type, use the following command to record a machine type for each instance:

      $nova-manage libvirt update_machine_type <instance_uuid> <machine_type>

      • Replace <instance_uuid> with the UUID of the instance.
      • Replace <machine_type> with the machine type originally used when the instance was created.

2. For each instance, show the machine type of the instance so you know what needs to be updated:

   $nova-manage libvirt get_machine_type <instance_uuid>

3. For each instance that needs to have its machine type changed to q35, complete the following steps:

   1. Test that the OS within the instance is compatible with the new hardware:

      1. Create a test instance.

      2. Verify that the test instance has a machine type other than q35:

         $nova-manage libvirt get_machine_type <test_instance_uuid>

      3. Stop the test instance:

         $openstack server stop <test_instance_uuid>

      4. Update the machine type of the test instance to q35:

         $nova-manage libvirt update_machine_type <test_instance_uuid> q35 --force

      5. Start the test instance:

         $openstack server start <test_instance_uuid>

      6. Verify that you can log in to the test instance and ensure that it is working as expected.

   2. In a test environment, verify that the OS within the instance can tolerate the change without re-creation:

      1. Stop the instance:

         $openstack server stop <instance_uuid>

      2. Update the machine type and use the --force value:

         $nova-manage libvirt update_machine_type <instance_uuid> q35 --force

      3. Update the disk controller bus and set hw_cdrom_bus to sata:

         $nova-manage image_property set <instance_uuid> --property hw_cdrom_bus=sata

      4. Start the instance:

         $openstack server start <instance_uuid>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Log in to a Controller node as the heat-admin user:

   (undercloud)$ metalsmith list
   $ ssh heat-admin@<controller_ip>

   Replace <controller_ip> with the IP address of the Controller node.

4. Retrieve the list of instances that have no machine type set:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt list_unset_machine_type

5. Check the NovaHWMachineType parameter in the nova-hw-machine-type-upgrade.yaml file for the default machine type for the instance host. The default value for the NovaHWMachineType parameter in RHOSP 16.2 is as follows:

   x86_64=pc-i440fx-rhel7.6.0,aarch64=virt-rhel7.6.0,ppc64=pseries-rhel7.6.0,ppc64le=pseries-rhel7.6.0

6. Update the system metadata of each instance with the default instance machine type:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt update_machine_type <instance_uuid> <machine_type>

   • Replace <instance_uuid> with the UUID of the instance.

   • Replace <machine_type> with the machine type to record for the instance.

     Warning

     If you set the machine type to something other than the machine type of the image on which the instance was booted, the existing instance might fail to boot.

7. Confirm that the machine type is recorded for all instances:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-status upgrade check

   This command returns a warning if an instance is found without a machine type. If you get this warning, repeat this procedure from step 4.

8. Change the default value of NovaHWMachineType in a Compute environment file to x86_64=q35 and deploy the overcloud.

Verification

1. Create an instance that has the default machine type:

   (overcloud)$ openstack server create --flavor <flavor> \
     --image <image> --network <network> \
     --wait defaultMachineTypeInstance

   • Replace <flavor> with the name or ID of a flavor for the instance.
   • Replace <image> with the name or ID of an image that does not set hw_machine_type.
   • Replace <network> with the name or ID of the network to connect the instance to.

2. Verify that the instance machine type is set to the default value:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt get_machine_type <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

3. Hard reboot an instance with a machine type of x86_64=pc-i440fx:

   (overcloud)$ openstack server reboot --hard <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

4. Verify that the instance machine type has not been changed:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt get_machine_type <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to re-enable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command.

4. If you use SBD fencing, reset the watchdog timer device interval to its original value before you disabled it:

   # pcs property set stonith-watchdog-timeout=<interval>

   • Replace <interval> with the original value of the watchdog timer device, for example, 10.
5. In the fencing.yaml environment file, set the EnableFencing parameter to true.

Procedure

1. Enter the shell for the horizon container:

   $ podman exec -it horizon /bin/bash

2. Navigate to the directory that contains the files to compress:

   $ cd /usr/bin/

3. Run the compression:

   $ python3 manage.py compress