Chapter 2. Director-Based Environments: Performing Updates to Minor Versions

The director relies on standard RPM methods to update your environment. This involves ensuring your director’s host uses the latest packages through yum.

The Undercloud update process might download new image archives from the rhosp-director-images and rhosp-director-images-ipa packages. Check the yum log to determine if new image archives are available:

$ sudo grep "rhosp-director-images" /var/log/yum.log

If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

$ rm -rf ~/images/*

Copy the new image archives:

$ cp /usr/share/rhosp-director-images/overcloud-full-latest-8.0.tar ~/images/.
$ cp /usr/share/rhosp-director-images/ironic-python-agent-latest-8.0.tar ~/images/.

Extract the archives:

$ cd ~/images
$ for tarfile in *.tar; do tar -xf $tarfile; done

Import the latest images into the director and configure nodes to use the new images

$ openstack overcloud image upload --update-existing --image-path ~/images/.
$ openstack baremetal configure boot

To finalize the image update, verify the existence of the new images:

$ openstack image list
$ ls -l /httpboot

The director is now updated and using the latest images. You do not need to restart any services after the update.

Due to a known issue (see BZ#1278181), the Overcloud requires some manual configuration of its configuration agent. This involves copying a new version of the configuration agent script from the director to each node in the Overcloud.

Log in as the stack user on the director host and source the Undercloud configuration:

$ source ~/stackrc

Copy the configuration agent (55-heat-config) to each Overcloud node. Use the following command to do this for all hosts:

$ for i in $(nova list | awk '/Running/ {print $(NF-1)}' | awk -F"=" '{print $NF}' ) ; do echo $i ; scp -o StrictHostKeyChecking=no /usr/share/openstack-heat-templates/software-config/elements/heat-config/os-refresh-config/configure.d/55-heat-config heat-admin@${i}: ; ssh -o StrictHostKeyChecking=no heat-admin@${i} 'sudo /bin/bash -c "cp /home/heat-admin/55-heat-config /usr/libexec/os-refresh-config/configure.d/55-heat-config"' ; done

This ensures the configuration agent is up-to-date.

This Overcloud also needs to recreate some post-deployment files. The director includes a script to achieve this. Copy and execute the heat-config-rebuild-deployed script on each node. Use the following command to do this for all nodes:

$ for i in $(nova list | awk '/Running/ {print $(NF-1)}' | awk -F"=" '{print $NF}') ; do echo $i ; scp  -o StrictHostKeyChecking=no /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed heat-admin@${i}: ; ssh -o StrictHostKeyChecking=no heat-admin@${i} 'sudo /bin/bash -c "mkdir -p /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin ; cp heat-config-rebuild-deployed /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed ; chmod +x /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed ; /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed"' ; done

The Overcloud relies on standard RPM methods to update the environment. This involves performing an update on all nodes using the openstack overcloud update from the director.

Use the -e to include environment files relevant to your Overcloud and its upgrade path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

Running an update on all nodes in parallel might cause problems. For example, an update of a package might involve restarting a service, which can disrupt other nodes. This is why the update process updates each node using a set of breakpoints. This means nodes are updated one by one. When one node completes the package update, the update process moves on to the next node. The update process also requires the -i option, which puts the command in an interactive mode that requires confirmation at each breakpoint. Without the -i option, the update remains paused at the first breakpoint.

The following is an example update command for updating the Overcloud:

$ openstack overcloud update stack overcloud -i \
  --templates  \
  -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /home/stack/templates/storage-environment.yaml \
  -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
  [-e <environment_file>|...]

Running this command starts the update process. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example:

not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-compute-0']
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode:

Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a Python-based regular expression to clear breakpoints on multiple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have completed their update.

The update command reports a COMPLETE status when the update completes:

...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
COMPLETE
update finished with status COMPLETE

If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes:

$ sudo pcs property set stonith-enabled=true