Chapter 18. Upgrading a standard overcloud

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to clipboard

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Copy to clipboard

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

   Copy to clipboard

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to clipboard

2. Identify the bootstrap Controller node by running the following command on the undercloud node:

   $ tripleo-ansible-inventory --list --stack overcloud |jq .overcloud_Controller.hosts[0]

   Copy to clipboard

3. Upgrade the bootstrap Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack <stack_name> --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      Copy to clipboard

      Replace <stack_name> with the name of your stack.

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      Copy to clipboard

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      Copy to clipboard

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      Copy to clipboard

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

   6. Verify that after the upgrade, the new Pacemaker cluster is started and that the control plane services such as galera, rabbit, haproxy, and redis are running:

      $ sudo pcs status

      Copy to clipboard

4. Upgrade the next Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      Copy to clipboard

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

      Copy to clipboard

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      Copy to clipboard

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      Copy to clipboard

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Verify that the old cluster is no longer running:

      $ sudo pcs status

      Copy to clipboard

      An error similar to the following is displayed when the cluster is not running:

      Error: cluster is not currently running on this node

      Copy to clipboard

   2. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      Copy to clipboard

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   3. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      Copy to clipboard

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to clipboard

2. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      Copy to clipboard

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Optional: If you use a Ceph subscription and have configured director to use the overcloud-minimal image for Ceph storage nodes, you must complete the following steps:

      1. Log in to the node and unset the Red Hat Enterprise Linux (RHEL) minor release version:

         $ sudo subscription-manager release --unset

         Copy to clipboard

      2. On the node, perform a system update:

         $ sudo dnf -y update

         Copy to clipboard

      3. Reboot the node:

         $ sudo reboot

         Copy to clipboard

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      Copy to clipboard

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      Copy to clipboard

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      Copy to clipboard

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      Copy to clipboard

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2

      Copy to clipboard

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      Copy to clipboard

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to clipboard
2. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   Copy to clipboard

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   Copy to clipboard

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Copy to clipboard

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Copy to clipboard

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to clipboard

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein
3. To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file.

4. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Copy to clipboard

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (fencing.yaml) with the EnableFencing parameter set to true.
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
5. Wait until the stack synchronization completes.