Upgrading Red Hat OpenStack Platform

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

   Copy to Clipboard Toggle word wrap

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

      Copy to Clipboard Toggle word wrap

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

      Copy to Clipboard Toggle word wrap
   3. Repeat these steps for all Controller and composable nodes.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node to reboot and log into it.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log in to a Ceph MON or Controller node and check the cluster status:

   $ sudo ceph -s

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the Ceph MON or Controller node, reboot the next Ceph Storage node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo ceph osd unset noout
   $ sudo ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo ceph status

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. To identify the Compute node that you intend to reboot, list all Compute nodes:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

3. From the overcloud, select a Compute Node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

   Copy to Clipboard Toggle word wrap

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host <hostname> --all-projects

   Copy to Clipboard Toggle word wrap
5. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

6. Log into the Compute Node and reboot it:

   [heat-admin@overcloud-compute-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
7. Wait until the node boots.

8. Enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname> nova-compute --enable

   Copy to Clipboard Toggle word wrap

9. Verify that the Compute node is enabled:

   (overcloud) $ openstack compute service list

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

   Copy to Clipboard Toggle word wrap

3. Check the undercloud free space:

   (undercloud) $ df -h

   Copy to Clipboard Toggle word wrap

   Use the "Undercloud Requirements" as a basis to determine if you have adequate free space.

4. If you have NTP installed on the undercloud, check that clocks are synchronized:

   (undercloud) $ sudo ntpstat

   Copy to Clipboard Toggle word wrap

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   Copy to Clipboard Toggle word wrap

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   Copy to Clipboard Toggle word wrap

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

   Copy to Clipboard Toggle word wrap

4. Check for failed containerized services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'exited=1' --all" ; done
   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'status=dead' -f 'status=restarting'" ; done

   Copy to Clipboard Toggle word wrap

5. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg'

   Copy to Clipboard Toggle word wrap

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | cut -d, -f 1,2,18,37,57 | column -s, -t

   Copy to Clipboard Toggle word wrap

   Replace <PASSWORD> and <IP ADDRESS> details with the actual details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

   Note

   In case the nodes run Redis services, only one node displays an ON status for that service. This is because Redis is an active-passive service, which runs only on one node at a time.

6. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec clustercheck clustercheck" ; done

   Copy to Clipboard Toggle word wrap

7. Check RabbitMQ cluster health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec $(ssh heat-admin@$NODE "sudo docker ps -f 'name=.*rabbitmq.*' -q") rabbitmqctl node_health_check" ; done

   Copy to Clipboard Toggle word wrap

8. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Copy to Clipboard Toggle word wrap

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

9. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

   Copy to Clipboard Toggle word wrap

10. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

    Copy to Clipboard Toggle word wrap

11. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

    Copy to Clipboard Toggle word wrap

12. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

    Copy to Clipboard Toggle word wrap

13. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

    Copy to Clipboard Toggle word wrap

14. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    Copy to Clipboard Toggle word wrap

    All agents should be Alive and their state should be UP.

15. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up

16. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up.

Procedure

1. Log in to director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-12-rpms

   Copy to Clipboard Toggle word wrap

3. Set the RHEL version to RHEL 7.9:

   $ sudo subscription-manager release --set=7.9

   Copy to Clipboard Toggle word wrap

4. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-13-rpms

   Copy to Clipboard Toggle word wrap

5. Re-enable updates to the overcloud base images:

   $ sudo yum-config-manager --setopt=exclude= --save

   Copy to Clipboard Toggle word wrap

6. Run yum to upgrade the director’s main packages:

   $ sudo yum update -y python-tripleoclient

   Copy to Clipboard Toggle word wrap

7. Run the following command to upgrade the undercloud:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
8. Wait until the undercloud upgrade process completes.

9. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
10. Wait until the node boots.

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

   Copy to Clipboard Toggle word wrap

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-13.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-13.0.tar; do tar -xvf $i; done
   $ cd ~

   Copy to Clipboard Toggle word wrap

4. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

   Copy to Clipboard Toggle word wrap

5. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

   Copy to Clipboard Toggle word wrap

6. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the openstack-tripleo-heat-templates-compat package:

   $ sudo yum install openstack-tripleo-heat-templates-compat

   Copy to Clipboard Toggle word wrap

   This installs the previous templates in the compat directory of your Heat template collection (/usr/share/openstack-tripleo-heat-templates/compat) and also creates a link to compat named after the previous version (pike). These templates are backwards compatible with the upgraded director, which means you can use the latest version of the director to install an overcloud of the previous version.

2. Create a temporary copy of the core Heat templates:

   $ cp -a /usr/share/openstack-tripleo-heat-templates /tmp/osp13

   Copy to Clipboard Toggle word wrap

3. Move the previous version into its own directory:

   $ mv /tmp/osp13/compat /tmp/osp12

   Copy to Clipboard Toggle word wrap

4. Perform a diff on the contents of both directories:

   $ diff -urN /tmp/osp12 /tmp/osp13

   Copy to Clipboard Toggle word wrap

   This shows the core template changes from one version to the next. These changes provide an idea of what should occur during the overcloud upgrade.

Procedure

1. To pull the images directly from registry.redhat.io in your overcloud deployment, an environment file is required to specify the image parameters. Run the following command to generate the container image environment file:

   (undercloud) $ sudo openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

2. Modify the overcloud_images.yaml file and include the following parameters to authenticate with registry.redhat.io during deployment:

   ContainerImageRegistryLogin: true
   ContainerImageRegistryCredentials:
     registry.redhat.io:
       <USERNAME>: <PASSWORD>

   Copy to Clipboard Toggle word wrap

   • Replace <USERNAME> and <PASSWORD> with your credentials for registry.redhat.io.

     The overcloud_images.yaml file contains the image locations on the undercloud. Include this file with your deployment.

     Note

     Before you run the openstack overcloud deploy command, you must log in to the remote registry:

     (undercloud) $ sudo docker login registry.redhat.io

     Copy to Clipboard Toggle word wrap

Procedure

1. Find the address of the local undercloud registry. The address uses the following pattern:

   <REGISTRY_IP_ADDRESS>:8787

   Copy to Clipboard Toggle word wrap

   Use the IP address of your undercloud, which you previously set with the local_ip parameter in your undercloud.conf file. For the commands below, the address is assumed to be 192.168.24.1:8787.

2. Log in to registry.redhat.io:

   (undercloud) $ docker login registry.redhat.io --username $RH_USER --password $RH_PASSWD

   Copy to Clipboard Toggle word wrap

3. Create a template to upload the images to the local registry, and the environment file to refer to those images:

   (undercloud) $ openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --push-destination=192.168.24.1:8787 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml \
     --output-images-file /home/stack/local_registry_images.yaml

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

4. Verify that the following two files have been created:

   • local_registry_images.yaml, which contains container image information from the remote source. Use this file to pull the images from the Red Hat Container Registry (registry.redhat.io) to the undercloud.
   • overcloud_images.yaml, which contains the eventual image locations on the undercloud. You include this file with your deployment.

5. Pull the container images from the remote registry and push them to the undercloud registry:

   (undercloud) $ openstack overcloud container image upload \
     --config-file  /home/stack/local_registry_images.yaml \
     --verbose

   Copy to Clipboard Toggle word wrap

   Pulling the required images might take some time depending on the speed of your network and your undercloud disk.

   Note

   The container images consume approximately 10 GB of disk space.

6. The images are now stored on the undercloud’s docker-distribution registry. To view the list of images on the undercloud’s docker-distribution registry, run the following command:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog | jq .repositories[]

   Copy to Clipboard Toggle word wrap
   Note

   The _catalog resource by itself displays only 100 images. To display more images, use the ?n=<interger> query string with the _catalog resource to display a larger number of images:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog?n=150 | jq .repositories[]

   Copy to Clipboard Toggle word wrap

   To view a list of tags for a specific image, use the skopeo command:

   (undercloud) $ curl -s http://192.168.24.1:8787/v2/rhosp13/openstack-keystone/tags/list | jq .tags

   Copy to Clipboard Toggle word wrap

   To verify a tagged image, use the skopeo command:

   (undercloud) $ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp13/openstack-keystone:13.0-44

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a template to pull images to the local registry:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud container image prepare \
     --namespace=rhosp13 \
     --prefix=openstack- \
     --output-images-file /home/stack/satellite_images

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.
   Note

   This version of the openstack overcloud container image prepare command targets the registry on the registry.redhat.io to generate an image list. It uses different values than the openstack overcloud container image prepare command used in a later step.

2. This creates a file called satellite_images with your container image information. You will use this file to synchronize container images to your Satellite 6 server.

3. Remove the YAML-specific information from the satellite_images file and convert it into a flat file containing only the list of images. The following sed commands accomplish this:

   (undercloud) $ awk -F ':' '{if (NR!=1) {gsub("[[:space:]]", ""); print $2}}' ~/satellite_images > ~/satellite_images_names

   Copy to Clipboard Toggle word wrap

   This provides a list of images that you pull into the Satellite server.

4. Copy the satellite_images_names file to a system that contains the Satellite 6 hammer tool. Alternatively, use the instructions in the Hammer CLI Guide to install the hammer tool to the undercloud.

5. Run the following hammer command to create a new product (OSP13 Containers) to your Satellite organization:

   $ hammer product create \
     --organization "ACME" \
     --name "OSP13 Containers"

   Copy to Clipboard Toggle word wrap

   This custom product will contain our images.

6. Add the base container image to the product:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhosp13/openstack-base \
     --name base

   Copy to Clipboard Toggle word wrap

7. Add the overcloud container images from the satellite_images file.

   $ while read IMAGE; do \
     IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \
     hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name $IMAGE \
     --name $IMAGENAME ; done < satellite_images_names

   Copy to Clipboard Toggle word wrap

8. Synchronize the container images:

   $ hammer product synchronize \
     --organization "ACME" \
     --name "OSP13 Containers"

   Copy to Clipboard Toggle word wrap

   Wait for the Satellite server to complete synchronization.

   Note

   Depending on your configuration, hammer might ask for your Satellite server username and password. You can configure hammer to automatically login using a configuration file. See the "Authentication" section in the Hammer CLI Guide.

9. If your Satellite 6 server uses content views, create a new content view version to incorporate the images.

10. Check the tags available for the base image:

    $ hammer docker tag list --repository "base" \
      --organization "ACME" \
      --product "OSP13 Containers"

    Copy to Clipboard Toggle word wrap

    This displays tags for the OpenStack Platform container images.

11. Return to the undercloud and generate an environment file for the images on your Satellite server. The following is an example command for generating the environment file:

    (undercloud) $ openstack overcloud container image prepare \
      --namespace=satellite6.example.com:5000 \
      --prefix=acme-osp13_containers- \
      --tag-from-label {version}-{release} \
      --output-env-file=/home/stack/templates/overcloud_images.yaml

    Copy to Clipboard Toggle word wrap
    Note

    This version of the openstack overcloud container image prepare command targets the Satellite server. It uses different values than the openstack overcloud container image prepare command used in a previous step.

    When running this command, include the following data:

    • --namespace - The URL and port of the registry on the Satellite server. The registry port on Red Hat Satellite is 5000. For example, --namespace=satellite6.example.com:5000.

      Note

      If you are using Red Hat Satellite version 6.10, you do not need to specify a port. The default port of 443 is used. For more information, see "How can we adapt RHOSP13 deployment to Red Hat Satellite 6.10?".

    • --prefix= - The prefix is based on a Satellite 6 convention for labels, which uses lower case characters and substitutes spaces for underscores. The prefix differs depending on whether you use content views:

      • If you use content views, the structure is [org]-[environment]-[content view]-[product]-. For example: acme-production-myosp13-osp13_containers-.
      • If you do not use content views, the structure is [org]-[product]-. For example: acme-osp13_containers-.
    • --tag-from-label {version}-{release} - Identifies the latest tag for each image.
    • -e - Include any environment files for optional services.
    • -r - Include a custom roles file.

    • --set ceph_namespace, --set ceph_image, --set ceph_tag - If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location. Note that ceph_image now includes a Satellite-specific prefix. This prefix is the same value as the --prefix option. For example:

      --set ceph_image=acme-osp13_containers-rhceph-3-rhel7

      Copy to Clipboard Toggle word wrap

      This ensures the overcloud uses the Ceph container image using the Satellite naming convention.

12. The overcloud_images.yaml file contains the image locations on the Satellite server. Include this file with your deployment.

Procedure

1. Edit the environment file containing your registration details. For example:

   $ vi ~/templates/rhel-registration/environment-rhel-registration.yaml

   Copy to Clipboard Toggle word wrap

2. Edit the following parameter values:

   rhel_reg_repos

   Update to include the new repositories for Red Hat OpenStack Platform 13.

   rhel_reg_activation_key

   Update the activation key to access the Red Hat OpenStack Platform 13 repositories.

   rhel_reg_sat_repo

   If using a newer version of Red Hat Satellite 6, update the repository containing Satellite 6’s management tools.

3. Save the environment file.

Procedure

1. Select an environment file and the check if it has an ExtraConfig parameter:

   $ grep ExtraConfig ~/templates/custom-config.yaml

   Copy to Clipboard Toggle word wrap
2. If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig) in the chosen file, check the full parameter structure in that file.

3. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value, it might have been been replaced with a parameter with an actual Puppet class. For example:

   parameter_defaults:
     ExtraConfig:
       neutron::config::dhcp_agent_config:
         'DEFAULT/dnsmasq_local_resolv':
           value: 'true'

   Copy to Clipboard Toggle word wrap

4. Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:

   $ grep dnsmasq_local_resolv

   Copy to Clipboard Toggle word wrap

   If so, change to the new interface.

5. The following are examples to demonstrate the change in syntax:

   • Example 1:

     parameter_defaults:
       ExtraConfig:
         neutron::config::dhcp_agent_config:
           'DEFAULT/dnsmasq_local_resolv':
             value: 'true'

     Copy to Clipboard Toggle word wrap

     Changes to:

     parameter_defaults:
       ExtraConfig:
         neutron::agents::dhcp::dnsmasq_local_resolv: true

     Copy to Clipboard Toggle word wrap

   • Example 2:

     parameter_defaults:
       ExtraConfig:
         ceilometer::config::ceilometer_config:
           'oslo_messaging_rabbit/rabbit_qos_prefetch_count':
             value: '32'

     Copy to Clipboard Toggle word wrap

     Changes to:

     parameter_defaults:
       ExtraConfig:
         oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'

     Copy to Clipboard Toggle word wrap

Procedure

1. Run the following commands to save a list of node IP addresses in the OVERCLOUD_HOSTS environment variable:

   $ source ~/stackrc
   $ export OVERCLOUD_HOSTS=$(openstack server list -f value -c Networks | cut -d "=" -f 2 | tr '\n' ' ')

   Copy to Clipboard Toggle word wrap

2. Run the following script:

   $ /usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/enable-ssh-admin.sh

   Copy to Clipboard Toggle word wrap

3. Proceed with the upgrade.

   • When using the openstack overcloud upgrade run command with pre-provisioned nodes, include the --ssh-user tripleo-admin parameter.

   • When upgrading Compute or Object Storage nodes, use the following:

     1. Use the -U option with the upgrade-non-controller.sh script and specify the stack user. This is because the default user for pre-provisioned nodes is stack and not heat-admin.

     2. Use the node’s IP address with the --upgrade option. This is because the nodes are not managed with the director’s Compute (nova) and Bare Metal (ironic) services and do not have a node name.

        For example:

        $ upgrade-non-controller.sh -U stack --upgrade 192.168.24.100

        Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --templates \
       -e /home/stack/templates/overcloud_images.yaml \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e)
   • The environment file with your new container image locations (-e). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
3. Wait until the upgrade preparation completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. If you use monolithic Controller nodes, run the upgrade command against the Controller role:

   $ openstack overcloud upgrade run --nodes Controller

   Copy to Clipboard Toggle word wrap
   • If you use a custom stack name, pass the name with the --stack option.

3. If you use Controller services split across multiple roles:

   1. Run the upgrade command for roles with Pacemaker services:

      $ openstack overcloud upgrade run --nodes ControllerOpenStack
      $ openstack overcloud upgrade run --nodes Database
      $ openstack overcloud upgrade run --nodes Messaging
      $ openstack overcloud upgrade run --nodes Telemetry

      Copy to Clipboard Toggle word wrap
      • If you use a custom stack name, pass the name with the --stack option.

   2. Run the upgrade command for the Networker role:

      $ openstack overcloud upgrade run --nodes Networker

      Copy to Clipboard Toggle word wrap
      • If you use a custom stack name, pass the name with the --stack option.

   3. Run the upgrade command for any remaining custom roles, except for Compute or CephStorage roles:

      $ openstack overcloud upgrade run --nodes ObjectStorage

      Copy to Clipboard Toggle word wrap
      • If you use a custom stack name, pass the name with the --stack option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade command:

   $ openstack overcloud upgrade run --nodes Compute

   Copy to Clipboard Toggle word wrap
   • If you are using a custom stack name, pass the name with the --stack option.
   • If you are using custom Compute roles, ensure that you include the role names with the --nodes option.
3. Wait until the Compute node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade command:

   $ openstack overcloud upgrade run --nodes CephStorage

   Copy to Clipboard Toggle word wrap
   • If using a custom stack name, pass the name with the --stack option.
3. Wait until the node upgrade completes.

4. Run the Ceph Storage upgrade command. For example:

   $ openstack overcloud ceph-upgrade run \
       --templates \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
       -e /home/stack/templates/ceph-customization.yaml \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e). For example:

     • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
     • The relevant environment files for your Ceph Storage nodes.
     • Any additional environment files relevant to your environment.
   • If using a custom stack name, pass the name with the --stack option.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
5. Wait until the Ceph Storage node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade command:

   $ openstack overcloud upgrade run --roles ComputeHCI

   Copy to Clipboard Toggle word wrap

   If you are using a custom stack name, pass the name to the upgrade command with the --stack option.

3. Run the Ceph Storage upgrade command. For example:

   $ openstack overcloud ceph-upgrade run \
       --templates \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
       -e /home/stack/templates/ceph-customization.yaml \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e). For example:

     • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
     • The relevant environment files for your Ceph Storage nodes.
   • If using a custom stack name, pass the name with the --stack option.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
4. Wait until the Ceph Storage node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade command for the Compute node:

   $ openstack overcloud upgrade run --roles Compute
   If using a custom stack name, pass the name with the --stack option.

   Copy to Clipboard Toggle word wrap
3. Wait until the node upgrade completes.

4. Run the upgrade command for the ComputeHCI node:

   $ openstack overcloud upgrade run --roles ComputeHCI
   If using a custom stack name, pass the name with the --stack option.

   Copy to Clipboard Toggle word wrap
5. Wait until the node upgrade completes.

6. Run the upgrade command for the Ceph Storage node:

   $ openstack overcloud upgrade run --roles CephStorage

   Copy to Clipboard Toggle word wrap
7. Wait until the Ceph Storage node upgrade completes.

8. Run the Ceph Storage upgrade command. For example:

   $ openstack overcloud ceph-upgrade run \
       --templates \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
       -e /home/stack/templates/ceph-customization.yaml \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e). For example:

     • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
     • The relevant environment files for your Ceph Storage nodes.
     • Any additional environment files relevant to your environment.
   • If using a custom stack name, pass the name with the --stack option.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
9. Wait until the Ceph Storage node upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --templates \
       -e <ENVIRONMENT FILE>

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e).
   • If using a custom stack name, pass the name with the --stack option.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
3. Wait until the upgrade finalization completes.