Chapter 4. Updating the Overcloud
This process updates the overcloud.
Prerequisites
- You have updated the undercloud to the latest version.
4.1. Speeding up the overcloud update Copy linkLink copied to clipboard!
To speed up the overcloud update process, you can configure the DockerPuppetProcessCount heat parameter, archive deleted database entries, and download the required packages on the overcloud nodes before you perform an update.
For more information about speeding up the update process for large OpenStack deployments, see the Red Hat Knowledgebase article Openstack Director Node Performance Tuning for large deployments.
Procedure
-
Log in to the undercloud as the
stackuser. Source the
stackrcfile:$ source ~/stackrcTo increase the number of concurrent processes that
container-puppetuses to generate configuration files, you must configure theDockerPuppetProcessCountparameter.Create an environment file called
updates-environment.yamlin yourtemplatesdirectory:$ touch ~/templates/updates-environment.yamlEdit the file and add the following content:
parameter_defaults: DockerPuppetProcessCount: 8-
Use the
-eoption to include this environment file when you run theopenstack overcloud update prepare,openstack overcloud ceph-upgrade run, andopenstack overcloud update convergecommands.
On a Controller node, archive your deleted database entries:
From the overcloud, list all instances of the Controller nodes:
$ source ~/overcloudrc $ openstack server listLog on to a Controller node that is running the
nova_api_croncontainer:ssh heat-admin@<controller_ip>-
Replace
<controller name or IP>with the IP address of a Controller node.
-
Replace
Archive deleted database entries:
$ sudo docker exec -u 42436 -ti nova_api_cron bash $ nova-manage db archive_deleted_rows --max_rows 1000 $ exit
To download all packages that are required for an update on all the overcloud nodes, complete the following steps:
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory ~/inventory.yamlCreate the following Ansible playbook:
$ cat > ~/yum-download-only.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: Pre-download all packages on all overcloud nodes shell: yum upgrade -y --downloadonly become: true EOFRun the
yum-download-only.yamlAnsible playbook:$ ansible-playbook \ -i ~/inventory.yaml \ -f 20 ~/yum-download-only.yaml \ --limit Controller,Compute,CephStorage
4.2. Consideration for custom roles Copy linkLink copied to clipboard!
Check the following values in your roles file if your deployment includes custom roles:
-
Compare your custom roles file with the latest files in the
/usr/share/openstack-tripleo-heat-templates/rolesdirectory. Add any new parameters from theRoleParametersDefaultsections for relevant roles for your environment to the equivalent roles in your custom roles file. If you use Data Plane Development Kit (DPDK) and are upgrading from 13.4 or lower, ensure that the roles that contain OVS-DPDK services also contain the following mandatory parameters:
RoleParametersDefault: VhostuserSocketGroup: "hugetlbfs" TunedProfileName: "cpu-paritioning" NovaLibvirtRxQueueSize: 1024 NovaLibvirtTxQueueSize: 1024
4.3. Running the overcloud update preparation Copy linkLink copied to clipboard!
The update requires running openstack overcloud update prepare command, which performs the following tasks:
- Updates the overcloud plan to OpenStack Platform 13
- Prepares the nodes for the update
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the update preparation command:
$ openstack overcloud update prepare \ --templates \ -r <ROLES DATA FILE> \ -n <NETWORK DATA FILE> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yaml \ -e <ENVIRONMENT FILE> \ -e <ENVIRONMENT FILE> \ --stack <STACK_NAME> ...Include the following options relevant to your environment:
-
Custom configuration environment files (
-e). -
The environment file with your new container image locations (
-e). Note that the update command might display a warning about using the--container-registry-file. You can ignore this warning as this option is deprecated in favor of using-efor the container image environment file. -
If you use your own custom roles, include your custom roles (
roles_data) file (-r). -
If you use custom networks, include your composable network (
network_data) file (-n). -
If you deploy a high availability cluster, include the
--ntp-serveroption in the update preparation command, or include theNtpServerparameter and value in your environment file. -
If the name of your overcloud stack is different to the default name
overcloud, include the--stackoption in the update preparation command and replace<STACK_NAME>with the name of your stack.
-
Custom configuration environment files (
- Wait until the update preparation completes.
4.4. Updating the Ceph Storage cluster Copy linkLink copied to clipboard!
This process updates the Ceph Storage cluster. The process involves running the openstack overcloud ceph-upgrade run command to perform an update to a Red Hat Ceph Storage 3 cluster.
The following combinations of Ansible with ceph-ansible are supported:
-
ansible-2.6withceph-ansible-3.2 -
ansible-2.4withceph-ansible-3.1
If your environment has ansible-2.6 with ceph-ansible-3.1, update ceph-ansible to the newest version:
# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
# subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms
# yum update ceph-ansible
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the Ceph Storage update command. For example:
$ openstack overcloud ceph-upgrade run \ --templates \ -e <ENVIRONMENT FILE> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yamlInclude the following options relevant to your environment:
-
Custom configuration environment files (
-e) -
The environment file with your container image locations (
-e). Note that the update command might display a warning about using the--container-registry-file. You can ignore this warning, as this option is deprecated in favor of using-efor the container image environment file. -
If applicable, your custom roles (
roles_data) file (--roles-file) -
If applicable, your composable network (
network_data) file (--networks-file)
-
Custom configuration environment files (
- Wait until the Ceph Storage node update completes.
If the Heat stack times out during the procedure, see the Red Hat Knowledgebase article During sequential update of Ceph nodes openstack overcloud ceph-upgrade run appears to time out.
4.5. Updating all Controller nodes Copy linkLink copied to clipboard!
To update the Controller nodes to the latest Red Hat OpenStack Platform (RHOSP) 13 version, include the --nodes Controller option in the openstack overcloud update run command. The --nodes Controller option restricts the update operations to the Controller nodes only.
- Warning
- If you use Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you update the Controller nodes in order to avoid bug BZ#1910842.
Prerequisites
If you use the load-balancing service (octavia) and want to update from a release earlier than RHOSP 13 z13 (8 October 2020 maintenance release), to avoid bug BZ#1927169, you must run the database migrations that upgrade the load-balancing service in the correct order. You must update the bootstrap Controller node, before you can update the rest of the control plane.
To identify your current maintenance release, run the following command:
$ cat /etc/rhosp-releaseOn the undercloud node, to identify the bootstrap Controller node, run the following command and replace
<any_controller_node_IP_address>with the IP address of any of the Controller nodes in your deployment:$ ssh heat-admin@<any_controller_node_IP_address> sudo hiera -c /etc/puppet/hiera.yaml octavia_api_short_bootstrap_node_nameOn the undercloud node, run the
openstack overcloud update runcommand to update the bootstrap Controller node:$ openstack overcloud update run --nodes <bootstrap_node_name>
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the update command:
$ openstack overcloud update run --nodes Controller- Wait until the Controller node update completes.
4.6. Updating all Compute nodes Copy linkLink copied to clipboard!
This process updates all Compute nodes to the latest OpenStack Platform 13 version. The process involves running the openstack overcloud update run command and including the --nodes Compute option to restrict operations to the Compute nodes only.
- Parallelization considerations
When you update a large number of Compute nodes, to improve performance, you can run the
openstack overcloud update runcommand with the--nodes Computeoption in parallel on batches of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:$ openstack overcloud update run --nodes 'Compute[0:19]' > update-compute-0-19.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[20:39]' > update-compute-20-39.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[40:59]' > update-compute-40-59.log 2>&1 & $ openstack overcloud update run --nodes 'Compute[60:79]' > update-compute-60-79.log 2>&1 &The
'Compute[0:19]','Compute[20:39]','Compute[40:59]', and'Compute[60:79]'way of partitioning the nodes space is random and you don’t have control over the order in which the nodes are updated in each batch.To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:
$ openstack overcloud update run --nodes <Compute0>,<Compute1>,<Compute2>,<Compute3>
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the update command:
$ openstack overcloud update run --nodes Compute- Wait until the Compute node update completes.
4.7. Updating all HCI Compute nodes Copy linkLink copied to clipboard!
This process updates the Hyperconverged Infrastructure (HCI) Compute nodes. The process involves running the openstack overcloud update run command and including the --nodes ComputeHCI option to restrict operations to the HCI nodes only.
- Warning
- If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the update command:
$ openstack overcloud update run --nodes ComputeHCI- Wait until the node update completes.
4.8. Updating all Ceph Storage nodes Copy linkLink copied to clipboard!
This process updates the Ceph Storage nodes. The process involves running the openstack overcloud update run command and including the --nodes CephStorage option to restrict operations to the Ceph Storage nodes only.
- Warning
- If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.
Procedure
Source the
stackrcfile:$ source ~/stackrcUpdate group nodes.
To update all nodes in a group:
$ openstack overcloud update run --nodes <GROUP_NAME>To update a single node in a group:
$ openstack overcloud update run --nodes <GROUP_NAME> [NODE_INDEX]NoteEnsure that you update all nodes if you choose to update nodes individually.
The index of the first node in a group is zero (0). For example, to update the first node in a group named
CephStorage, the command is:openstack overcloud update run --nodes CephStorage[0]- Wait until the node update completes.
4.9. Finalizing the update Copy linkLink copied to clipboard!
The update requires a final step to update the overcloud stack. This ensures that the stack resource structure aligns with a regular deployment of Red Hat OpenStack Platform 13 and you can perform standard openstack overcloud deploy functions in the future.
Procedure
Source the
stackrcfile:$ source ~/stackrcRun the update finalization command:
$ openstack overcloud update converge \ --templates \ --stack <STACK_NAME> \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/updates-environment.yaml \ -e <ENVIRONMENT FILE> ...Include the following options relevant to your environment:
-
Custom configuration environment files (
-e) -
The environment file with your new container image locations (
-e). Note that the update command might display a warning about using the--container-registry-file. You can ignore this warning as this option is deprecated in favor of using-efor the container image environment file. -
If applicable, your custom roles (
roles_data) file (--roles-file) -
If applicable, your composable network (
network_data) file (--networks-file) -
If the name of your overcloud stack is different than the default name
overcloud, you must include the--stackoption in the update preparation command and replace<STACK_NAME>with the name of your overcloud stack.
-
Custom configuration environment files (
- Wait until the update finalization completes.