Chapter 4. Updating the Overcloud
This process updates the overcloud.
Prerequisites
- You have updated the undercloud to the latest version.
4.1. Speeding up the overcloud update
To speed up the overcloud update process, you can configure the DockerPuppetProcessCount
heat parameter, archive deleted database entries, and download the required packages on the overcloud nodes before you perform an update.
For more information about speeding up the update process for large OpenStack deployments, see the Red Hat Knowledgebase article Openstack Director Node Performance Tuning for large deployments.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
To increase the number of concurrent processes that
container-puppet
uses to generate configuration files, you must configure theDockerPuppetProcessCount
parameter.Create an environment file called
updates-environment.yaml
in yourtemplates
directory:$ touch ~/templates/updates-environment.yaml
Edit the file and add the following content:
parameter_defaults: DockerPuppetProcessCount: 8
-
Use the
-e
option to include this environment file when you run theopenstack overcloud update prepare
,openstack overcloud ceph-upgrade run
, andopenstack overcloud update converge
commands.
On a Controller node, archive your deleted database entries:
From the overcloud, list all instances of the Controller nodes:
$ source ~/overcloudrc $ openstack server list
Log on to a Controller node that is running the
nova_api_cron
container:ssh heat-admin@<controller_ip>
-
Replace
<controller name or IP>
with the IP address of a Controller node.
-
Replace
Archive deleted database entries:
$ sudo docker exec -u 42436 -ti nova_api_cron bash $ nova-manage db archive_deleted_rows --max_rows 1000 $ exit
To download all packages that are required for an update on all the overcloud nodes, complete the following steps:
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory ~/inventory.yaml
Create the following Ansible playbook:
$ cat > ~/yum-download-only.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: Pre-download all packages on all overcloud nodes shell: yum upgrade -y --downloadonly become: true EOF
Run the
yum-download-only.yaml
Ansible playbook:$ ansible-playbook \ -i ~/inventory.yaml \ -f 20 ~/yum-download-only.yaml \ --limit Controller,Compute,CephStorage
4.2. Consideration for custom roles
Check the following values in your roles file if your deployment includes custom roles:
-
Compare your custom roles file with the latest files in the
/usr/share/openstack-tripleo-heat-templates/roles
directory. Add any new parameters from theRoleParametersDefault
sections for relevant roles for your environment to the equivalent roles in your custom roles file. If you use Data Plane Development Kit (DPDK) and are upgrading from 13.4 or lower, ensure that the roles that contain OVS-DPDK services also contain the following mandatory parameters:
RoleParametersDefault: VhostuserSocketGroup: "hugetlbfs" TunedProfileName: "cpu-paritioning" NovaLibvirtRxQueueSize: 1024 NovaLibvirtTxQueueSize: 1024
4.3. Running the overcloud update preparation
The update requires running openstack overcloud update prepare
command, which performs the following tasks:
- Updates the overcloud plan to OpenStack Platform 13
- Prepares the nodes for the update
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update preparation command:
$ openstack overcloud update prepare \ --templates \ -r <ROLES DATA FILE> \ -n <NETWORK DATA FILE> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yaml \ -e <ENVIRONMENT FILE> \ -e <ENVIRONMENT FILE> \ --stack <STACK_NAME> ...
Include the following options relevant to your environment:
-
Custom configuration environment files (
-e
). -
The environment file with your new container image locations (
-e
). Note that the update command might display a warning about using the--container-registry-file
. You can ignore this warning as this option is deprecated in favor of using-e
for the container image environment file. -
If you use your own custom roles, include your custom roles (
roles_data
) file (-r
). -
If you use custom networks, include your composable network (
network_data
) file (-n
). -
If you deploy a high availability cluster, include the
--ntp-server
option in the update preparation command, or include theNtpServer
parameter and value in your environment file. -
If the name of your overcloud stack is different to the default name
overcloud
, include the--stack
option in the update preparation command and replace<STACK_NAME>
with the name of your stack.
-
Custom configuration environment files (
- Wait until the update preparation completes.
4.4. Updating the Ceph Storage cluster
This process updates the Ceph Storage cluster. The process involves running the openstack overcloud ceph-upgrade run
command to perform an update to a Red Hat Ceph Storage 3 cluster.
The following combinations of Ansible with ceph-ansible
are supported:
-
ansible-2.6
withceph-ansible-3.2
-
ansible-2.4
withceph-ansible-3.1
If your environment has ansible-2.6
with ceph-ansible-3.1
, update ceph-ansible
to the newest version:
# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms # subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms # yum update ceph-ansible
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the Ceph Storage update command. For example:
$ openstack overcloud ceph-upgrade run \ --templates \ -e <ENVIRONMENT FILE> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yaml
Include the following options relevant to your environment:
-
Custom configuration environment files (
-e
) -
The environment file with your container image locations (
-e
). Note that the update command might display a warning about using the--container-registry-file
. You can ignore this warning, as this option is deprecated in favor of using-e
for the container image environment file. -
If applicable, your custom roles (
roles_data
) file (--roles-file
) -
If applicable, your composable network (
network_data
) file (--networks-file
)
-
Custom configuration environment files (
- Wait until the Ceph Storage node update completes.
If the Heat stack times out during the procedure, see the Red Hat Knowledgebase article During sequential update of Ceph nodes openstack overcloud ceph-upgrade run
appears to time out.
4.5. Updating all Controller nodes
To update the Controller nodes to the latest Red Hat OpenStack Platform (RHOSP) 13 version, include the --nodes Controller
option in the openstack overcloud update run
command. The --nodes Controller
option restricts the update operations to the Controller nodes only.
- Warning
- If you use Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you update the Controller nodes in order to avoid bug BZ#1910842.
Prerequisites
If you use the load-balancing service (octavia) and want to update from a release earlier than RHOSP 13 z13 (8 October 2020 maintenance release), to avoid bug BZ#1927169, you must run the database migrations that upgrade the load-balancing service in the correct order. You must update the bootstrap Controller node, before you can update the rest of the control plane.
To identify your current maintenance release, run the following command:
$ cat /etc/rhosp-release
On the undercloud node, to identify the bootstrap Controller node, run the following command and replace
<any_controller_node_IP_address>
with the IP address of any of the Controller nodes in your deployment:$ ssh heat-admin@<any_controller_node_IP_address> sudo hiera -c /etc/puppet/hiera.yaml octavia_api_short_bootstrap_node_name
On the undercloud node, run the
openstack overcloud update run
command to update the bootstrap Controller node:$ openstack overcloud update run --nodes <bootstrap_node_name>
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --nodes Controller
- Wait until the Controller node update completes.
4.6. Updating all Compute nodes
This process updates all Compute nodes to the latest OpenStack Platform 13 version. The process involves running the openstack overcloud update run
command and including the --nodes Compute
option to restrict operations to the Compute nodes only.
- Parallelization considerations
When you update a large number of Compute nodes, to improve performance, you can run the
openstack overcloud update run
command with the--nodes Compute
option in parallel on batches of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:$ openstack overcloud update run --nodes 'Compute[0:19]' > update-compute-0-19.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[20:39]' > update-compute-20-39.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[40:59]' > update-compute-40-59.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[60:79]' > update-compute-60-79.log 2>&1 &
The
'Compute[0:19]'
,'Compute[20:39]'
,'Compute[40:59]'
, and'Compute[60:79]'
way of partitioning the nodes space is random and you don’t have control over the order in which the nodes are updated in each batch.To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:
$ openstack overcloud update run --nodes <Compute0>,<Compute1>,<Compute2>,<Compute3>
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --nodes Compute
- Wait until the Compute node update completes.
4.7. Updating all HCI Compute nodes
This process updates the Hyperconverged Infrastructure (HCI) Compute nodes. The process involves running the openstack overcloud update run
command and including the --nodes ComputeHCI
option to restrict operations to the HCI nodes only.
- Warning
- If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --nodes ComputeHCI
- Wait until the node update completes.
4.8. Updating all Ceph Storage nodes
This process updates the Ceph Storage nodes. The process involves running the openstack overcloud update run
command and including the --nodes CephStorage
option to restrict operations to the Ceph Storage nodes only.
- Warning
- If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Update group nodes.
To update all nodes in a group:
$ openstack overcloud update run --nodes <GROUP_NAME>
To update a single node in a group:
$ openstack overcloud update run --nodes <GROUP_NAME> [NODE_INDEX]
NoteEnsure that you update all nodes if you choose to update nodes individually.
The index of the first node in a group is zero (0). For example, to update the first node in a group named
CephStorage
, the command is:openstack overcloud update run --nodes CephStorage[0]
- Wait until the node update completes.
4.9. Finalizing the update
The update requires a final step to update the overcloud stack. This ensures that the stack resource structure aligns with a regular deployment of Red Hat OpenStack Platform 13 and you can perform standard openstack overcloud deploy
functions in the future.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update finalization command:
$ openstack overcloud update converge \ --templates \ --stack <STACK_NAME> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yaml \ -e <ENVIRONMENT FILE> ...
Include the following options relevant to your environment:
-
Custom configuration environment files (
-e
) -
The environment file with your new container image locations (
-e
). Note that the update command might display a warning about using the--container-registry-file
. You can ignore this warning as this option is deprecated in favor of using-e
for the container image environment file. -
If applicable, your custom roles (
roles_data
) file (--roles-file
) -
If applicable, your composable network (
network_data
) file (--networks-file
) -
If the name of your overcloud stack is different than the default name
overcloud
, you must include the--stack
option in the update preparation command and replace<STACK_NAME>
with the name of your overcloud stack.
-
Custom configuration environment files (
- Wait until the update finalization completes.