Fast Forward Upgrades

Procedure

1. Log in to the node that is running the ceph-mon service. This node is usually a Controller node or a standalone Ceph Monitor node.

2. View the status of the Ceph cluster:

   $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

   Copy to Clipboard Toggle word wrap
3. Confirm that the health status of the cluster is HEALTH_OK and that all of the Object Storage Daemons (OSD) are active.

Procedure

1. Log into the undercloud as the root user.

2. Back up the database:

   [root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql

   Copy to Clipboard Toggle word wrap

3. Create a backup directory and change the user ownership of the directory to the stack user:

   [root@director ~]# mkdir /backup
   [root@director ~]# chown stack: /backup

   Copy to Clipboard Toggle word wrap

   You will use this directory to store the archive containing the undercloud database and file system.

4. Change to the backup directory

   [root@director ~]# cd /backup

   Copy to Clipboard Toggle word wrap

5. Archive the database backup and the configuration files:

   [root@director ~]# tar --xattrs --xattrs-include='*.*' --ignore-failed-read -cf \
       undercloud-backup-$(date +%F).tar \
       /root/undercloud-all-databases.sql \
       /etc \
       /var/log \
       /var/lib/glance \
       /var/lib/certmonger \
       /var/lib/docker \
       /var/lib/registry \
       /srv/node \
       /root \
       /home/stack

   Copy to Clipboard Toggle word wrap
   • The --ignore-failed-read option skips any directory that does not apply to your undercloud.
   • The --xattrs and --xattrs-include='.' options include extended attributes, which are required to store metadata for Object Storage (swift) and SELinux.

   This creates a file named undercloud-backup-<date>.tar.gz, where <date> is the system date. Copy this tar file to a secure location.

Procedure

1. Perform the database backup:

   1. Log into a Controller node. You can access the overcloud from the undercloud:

      $ ssh heat-admin@192.0.2.100

      Copy to Clipboard Toggle word wrap

   2. Change to the root user:

      $ sudo -i

      Copy to Clipboard Toggle word wrap

   3. Create a temporary directory to store the backups:

      # mkdir -p /var/tmp/mysql_backup/

      Copy to Clipboard Toggle word wrap

   4. Obtain the database password and store it in the MYSQLDBPASS environment variable. The password is stored in the mysql::server::root_password variable within the /etc/puppet/hieradata/service_configs.json file. Use the following command to store the password:

      # MYSQLDBPASS=$(sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

      Copy to Clipboard Toggle word wrap

   5. Backup the database:

      # mysql -uroot -p$MYSQLDBPASS -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot -p$MYSQLDBPASS --single-transaction --databases > /var/tmp/mysql_backup/openstack_databases-$(date +%F)-$(date +%T).sql

      Copy to Clipboard Toggle word wrap

      This dumps a database backup called /var/tmp/mysql_backup/openstack_databases-<date>.sql where <date> is the system date and time. Copy this database dump to a secure location.

   6. Backup all the users and permissions information:

      # mysql -uroot -p$MYSQLDBPASS -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -uroot -p$MYSQLDBPASS -s -N -e | sed 's/$/;/' > /var/tmp/mysql_backup/openstack_databases_grants-$(date +%F)-$(date +%T).sql

      Copy to Clipboard Toggle word wrap

      This dumps a database backup called /var/tmp/mysql_backup/openstack_databases_grants-<date>.sql where <date> is the system date and time. Copy this database dump to a secure location.

2. Backup the Pacemaker configuration:

   1. Log into a Controller node.

   2. Run the following command to create an archive of the current Pacemaker configuration:

      # sudo pcs config backup pacemaker_controller_backup

      Copy to Clipboard Toggle word wrap
   3. Copy the resulting archive (pacemaker_controller_backup.tar.bz2) to a secure location.

3. Backup the OpenStack Telemetry database:

   1. Connect to any controller and get the IP of the MongoDB primary instance:

      # MONGOIP=$(sudo hiera -c /etc/puppet/hiera.yaml mongodb::server::bind_ip)

      Copy to Clipboard Toggle word wrap

   2. Create the backup:

      # mkdir -p /var/tmp/mongo_backup/
      # mongodump --oplog --host $MONGOIP --out /var/tmp/mongo_backup/

      Copy to Clipboard Toggle word wrap
   3. Copy the database dump in /var/tmp/mongo_backup/ to a secure location.

4. Backup the Redis cluster:

   1. Obtain the Redis endpoint from HAProxy:

      # REDISIP=$(sudo hiera -c /etc/puppet/hiera.yaml redis_vip)

      Copy to Clipboard Toggle word wrap

   2. Obtain the master password for the Redis cluster:

      # REDISPASS=$(sudo hiera -c /etc/puppet/hiera.yaml redis::masterauth)

      Copy to Clipboard Toggle word wrap

   3. Check connectivity to the Redis cluster:

      # redis-cli -a $REDISPASS -h $REDISIP ping

      Copy to Clipboard Toggle word wrap

   4. Dump the Redis database:

      # redis-cli -a $REDISPASS -h $REDISIP bgsave

      Copy to Clipboard Toggle word wrap

      This stores the database backup in the default /var/lib/redis/ directory. Copy this database dump to a secure location.

5. Backup the filesystem on each Controller node:

   1. Create a directory for the backup:

      # mkdir -p /var/tmp/filesystem_backup/

      Copy to Clipboard Toggle word wrap

   2. Run the following tar command:

      # tar --acls --ignore-failed-read --xattrs --xattrs-include='*.*' \
          -zcvf /var/tmp/filesystem_backup/`hostname`-filesystem-`date '+%Y-%m-%d-%H-%M-%S'`.tar \
          /etc \
          /srv/node \
          /var/log \
          /var/lib/nova \
          --exclude /var/lib/nova/instances \
          /var/lib/glance \
          /var/lib/keystone \
          /var/lib/cinder \
          /var/lib/heat \
          /var/lib/heat-config \
          /var/lib/heat-cfntools \
          /var/lib/rabbitmq \
          /var/lib/neutron \
          /var/lib/haproxy \
          /var/lib/openvswitch \
          /var/lib/redis \
          /var/lib/os-collect-config \
          /usr/libexec/os-apply-config \
          /usr/libexec/os-refresh-config \
          /home/heat-admin

      Copy to Clipboard Toggle word wrap

      The --ignore-failed-read option ignores any missing directories, which is useful if certain services are not used or separated on their own custom roles.

   3. Copy the resulting tar file to a secure location.

6. Archive deleted rows on the overcloud:

   1. Check for archived deleted instances:

      $ source ~/overcloudrc
      $ nova list --all-tenants --deleted

      Copy to Clipboard Toggle word wrap

   2. If there are no archived deleted instances, then archive the deleted instances by entering the following command on one of the overcloud Controller nodes:

      # su - nova -s /bin/bash -c "nova-manage --debug db archive_deleted_rows --max_rows 1000"

      Copy to Clipboard Toggle word wrap

      Rerun this command until you have archived all deleted instances.

   3. Purge all the archived deleted instances by entering the following command on one of the overcloud Controller nodes:

      # su - nova -s /bin/bash -c "nova-manage --debug db purge --all --all-cells"

      Copy to Clipboard Toggle word wrap

   4. Verify that there are no remaining archived deleted instances:

      $ nova list --all-tenants --deleted

      Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. Stop the main OpenStack Platform services:

   $ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

   Copy to Clipboard Toggle word wrap
   Note

   This causes a short period of downtime for the undercloud. The overcloud is still functional during the undercloud upgrade.

3. Set the RHEL version to RHEL 7.7:

   $ sudo subscription-manager release --set=7.7

   Copy to Clipboard Toggle word wrap

4. Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update:

   $ sudo yum update -y python-tripleoclient

   Copy to Clipboard Toggle word wrap

5. Run the openstack undercloud upgrade command:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
6. Wait until the command completes its execution.

7. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
8. Wait until the node boots.
9. Log into the undercloud as the stack user.

Procedure

1. Change the vhost user socket directory in a custom environment file, for example, network-environment.yaml:

   parameter_defaults:
     NeutronVhostuserSocketDir: "/var/lib/vhost_sockets"

   Copy to Clipboard Toggle word wrap

2. Add the ovs-dpdk-permissions.yaml file to your openstack overcloud deploy command to configure the qemu group setting as hugetlbfs for OVS-DPDK:

    -e environments/ovs-dpdk-permissions.yaml

   Copy to Clipboard Toggle word wrap
3. Ensure that vHost user ports for all instances are in dpdkvhostuserclient mode. For more information see Manually changing the vhost user port mode.

Procedure

1. Check the yum log to determine if new image archives are available:

   $ sudo grep "rhosp-director-images" /var/log/yum.log

   Copy to Clipboard Toggle word wrap

2. If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

   Copy to Clipboard Toggle word wrap

3. On the undercloud node, source the undercloud credentials:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

4. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done

   Copy to Clipboard Toggle word wrap

5. Import the latest images in to director and configure nodes to use the new images:

   $ cd ~/images
   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/
   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

   Copy to Clipboard Toggle word wrap

6. To finalize the image update, verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

   Copy to Clipboard Toggle word wrap

   Director also retains the old images and renames them using the timestamp of when they were updated. If you no longer need these images, delete them.

Procedure

1. Check your subscription management configuration for the rhel_reg_release parameter. If this parameter is not set, you must include it and set it version 7.7:

   parameter_defaults:
     ...
     rhel_reg_release: "7.7"

   Copy to Clipboard Toggle word wrap

   Ensure that you save the changes to the overcloud subscription management environment file.

2. Update the current plan using your original openstack overcloud deploy command and including the --update-plan-only option. For example:

   $ openstack overcloud deploy --update-plan-only \
     --templates  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/storage-environment.yaml \
     -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
     [-e <environment_file>|...]

   Copy to Clipboard Toggle word wrap

   The --update-plan-only only updates the Overcloud plan stored in the director. Use the -e option to include environment files relevant to your Overcloud and its update path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

   • Any network isolation files, including the initialization file (environments/network-isolation.yaml) from the heat template collection and then your custom NIC configuration file.
   • Any external load balancing environment files.
   • Any storage environment files.
   • Any environment files for Red Hat CDN or Satellite registration.
   • Any other custom environment files.

3. Create a static inventory file of your overcloud:

   $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

   Copy to Clipboard Toggle word wrap

   If you use an overcloud name different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

4. Create a playbook that contains a task to set the operating system version to Red Hat Enterprise Linux 7.7 on all nodes:

   $ cat > ~/set_release.yaml <<'EOF'
   - hosts: all
     gather_facts: false
     tasks:
       - name: set release to 7.7
         command: subscription-manager release --set=7.7
         become: true
   EOF

   Copy to Clipboard Toggle word wrap

5. Run the set_release.yaml playbook:

   $ ansible-playbook -i ~/inventory.yaml -f 25 ~/set_release.yaml --limit undercloud,Controller,Compute

   Copy to Clipboard Toggle word wrap

   Use the --limit option to apply the content to all Red Hat OpenStack Platform nodes.

6. Perform a package update on all nodes using the openstack overcloud update command:

   $ openstack overcloud update stack -i overcloud

   Copy to Clipboard Toggle word wrap

   The -i runs an interactive mode to update each node sequentially. When the update process completes a node update, the script provides a breakpoint for you to confirm. Without the -i option, the update remains paused at the first breakpoint. Therefore, it is mandatory to include the -i option.

   The script performs the following functions:

   1. The script runs on nodes one-by-one:

      1. For Controller nodes, this means a full package update.
      2. For other nodes, this means an update of Puppet modules only.

   2. Puppet runs on all nodes at once:

      1. For Controller nodes, the Puppet run synchronizes the configuration.
      2. For other nodes, the Puppet run updates the rest of the packages and synchronizes the configuration.

7. The update process starts. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example:

   starting package update on stack overcloud
   IN_PROGRESS
   IN_PROGRESS
   WAITING
   on_breakpoint: [u'overcloud-compute-0', u'overcloud-controller-2', u'overcloud-controller-1', u'overcloud-controller-0']
   Breakpoint reached, continue? Regexp or Enter=proceed (will clear 49913767-e2dd-4772-b648-81e198f5ed00), no=cancel update, C-c=quit interactive mode:

   Copy to Clipboard Toggle word wrap

   Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node.

8. The script automatically predefines the update order of nodes:

   • Each Controller node individually
   • Each individual Compute node individually
   • Each Ceph Storage node individually
   • All other nodes individually

   It is recommended to use this order to ensure a successful update, specifically:

   1. Clear the breakpoint of each Controller node individually. Each Controller node requires an individual package update in case the node’s services must restart after the update. This reduces disruption to highly available services on other Controller nodes.
   2. After the Controller node update, clear the breakpoints for each Compute node. You can also type a Compute node name to clear a breakpoint on a specific node or use a Python-based regular expression to clear breakpoints on multiple Compute nodes at once.
   3. Clear the breakpoints for each Ceph Storage nodes. You can also type a Ceph Storage node name to clear a breakpoint on a specific node or use a Python-based regular expression to clear breakpoints on multiple Ceph Storage nodes at once.
   4. Clear any remaining breakpoints to update the remaining nodes. You can also type a node name to clear a breakpoint on a specific node or use a Python-based regular expression to clear breakpoints on multiple nodes at once.
   5. Wait until all nodes have completed their update.

9. The update command reports a COMPLETE status when the update completes:

   ...
   IN_PROGRESS
   IN_PROGRESS
   IN_PROGRESS
   COMPLETE
   update finished with status COMPLETE

   Copy to Clipboard Toggle word wrap

10. If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, re-enable fencing with the following command on one of the Controller nodes:

    $ sudo pcs property set stonith-enabled=true

    Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

   Copy to Clipboard Toggle word wrap

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

      Copy to Clipboard Toggle word wrap

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

      Copy to Clipboard Toggle word wrap
   3. Repeat these steps for all Controller and composable nodes.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node to reboot and log into it.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log in to a Ceph MON or Controller node and check the cluster status:

   $ sudo ceph -s

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the Ceph MON or Controller node, reboot the next Ceph Storage node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo ceph osd unset noout
   $ sudo ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo ceph status

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. To identify the Compute node that you intend to reboot, list all Compute nodes:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

3. From the overcloud, select a Compute Node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

   Copy to Clipboard Toggle word wrap

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host <hostname> --all-projects

   Copy to Clipboard Toggle word wrap
5. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

6. Log into the Compute Node and reboot it:

   [heat-admin@overcloud-compute-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
7. Wait until the node boots.

8. Enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname> nova-compute --enable

   Copy to Clipboard Toggle word wrap

9. Verify that the Compute node is enabled:

   (overcloud) $ openstack compute service list

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into a node.

2. Run yum to check the system packages:

   $ sudo yum list qemu-img-rhev qemu-kvm-common-rhev qemu-kvm-rhev qemu-kvm-tools-rhev openvswitch

   Copy to Clipboard Toggle word wrap

3. Run ovs-vsctl to check the version currently running:

   $ sudo ovs-vsctl --version

   Copy to Clipboard Toggle word wrap
4. Repeat these steps for all nodes.

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check for failed Systemd services:

   $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

   Copy to Clipboard Toggle word wrap

3. Check the undercloud free space:

   $ df -h

   Copy to Clipboard Toggle word wrap

   Use the "Undercloud Requirements" as a basis to determine if you have adequate free space.

4. If you have NTP installed on the undercloud, check the clock is synchronized:

   $ sudo ntpstat

   Copy to Clipboard Toggle word wrap

5. Check the undercloud network services:

   $ openstack network agent list

   Copy to Clipboard Toggle word wrap

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   $ openstack compute service list

   Copy to Clipboard Toggle word wrap

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check the status of your bare metal nodes:

   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

   Copy to Clipboard Toggle word wrap

4. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication information for the haproxy.stats service:

   $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /etc/haproxy/haproxy.cfg'

   Copy to Clipboard Toggle word wrap

5. Use the connection and authentication information obtained from the previous step to check the connection status of RHOSP services.

   If SSL is not enabled, use these details in the following cURL request:

   $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

   Copy to Clipboard Toggle word wrap

   If SSL is enabled, use these details in the following cURL request:

   curl -s -u admin:<PASSWORD> "https://<HOSTNAME>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

   Copy to Clipboard Toggle word wrap

   Replace the <PASSWORD> and <IP ADDRESS> or <HOSTNAME> values with the respective information from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

6. Check overcloud database replication health:

   $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo clustercheck" ; done

   Copy to Clipboard Toggle word wrap

7. Check RabbitMQ cluster health:

   $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo rabbitmqctl node_health_check" ; done

   Copy to Clipboard Toggle word wrap

8. Check Pacemaker resource health:

   $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Copy to Clipboard Toggle word wrap

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

9. Check the disk space on each overcloud node:

   $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

   Copy to Clipboard Toggle word wrap

10. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

    $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

    Copy to Clipboard Toggle word wrap

11. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

    Copy to Clipboard Toggle word wrap
    Important

    The number of placement groups (PGs) for each Ceph object storage daemon (OSD) must not exceed 250 by default. Upgrading Ceph nodes with more PGs per OSD results in a warning state and might fail the upgrade process. You can increase the number of PGs per OSD before you start the upgrade process. For more information about diagnosing and troubleshooting this issue, see the article OpenStack FFU from 10 to 13 times out because Ceph PGs allocated in one or more OSDs is higher than 250.

12. Check that clocks are synchronized on overcloud nodes

    $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

    Copy to Clipboard Toggle word wrap

13. Source the overcloud access details:

    $ source ~/overcloudrc

    Copy to Clipboard Toggle word wrap

14. Check the overcloud network services:

    $ openstack network agent list

    Copy to Clipboard Toggle word wrap

    All agents should be Alive and their state should be UP.

15. Check the overcloud compute services:

    $ openstack compute service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up

16. Check the overcloud volume services:

    $ openstack volume service list

    Copy to Clipboard Toggle word wrap

    All agents' status should be enabled and their state should be up.

1. Source the overcloud access details:

   $ source ~/overcloudrc

   Copy to Clipboard Toggle word wrap

2. Find the server ID for the instance you want to take a snapshot of:

   $ openstack server list

   Copy to Clipboard Toggle word wrap

3. Shut down the source instance before you take the snapshot to ensure that all data is flushed to disk:

   $ openstack server stop SERVER_ID

   Copy to Clipboard Toggle word wrap

4. Create the snapshot image of the instance:

   $ openstack image create --id SERVER_ID SNAPSHOT_NAME

   Copy to Clipboard Toggle word wrap

5. Boot a new instance with this snapshot image:

   $ openstack server create --flavor DPDK_FLAVOR --nic net-id=DPDK_NET_ID--image SNAPSHOT_NAME INSTANCE_NAME

   Copy to Clipboard Toggle word wrap

6. Optionally, verify that the new instance status is ACTIVE:

   $ openstack server list

   Copy to Clipboard Toggle word wrap

Procedure

1. On each node, run the following command to save the entire yum history of the node in a file:

   $ sudo yum history list all > /home/heat-admin/$(hostname)-yum-history-all

   Copy to Clipboard Toggle word wrap

2. On each node, run the following command to save the ID of the last yum history item:

   $ sudo yum history list all | head -n 5 | tail -n 1 | awk '{print $1}' > /home/heat-admin/$(hostname)-yum-history-all-last-id

   Copy to Clipboard Toggle word wrap
3. Copy these files to a secure location.

Procedure

1. Log in to director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-10-rpms

   Copy to Clipboard Toggle word wrap

3. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-11-rpms

   Copy to Clipboard Toggle word wrap

4. Disable updates to the overcloud base images:

   $ sudo yum-config-manager --setopt=exclude=rhosp-director-images* --save

   Copy to Clipboard Toggle word wrap

5. Stop the main OpenStack Platform services:

   $ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

   Copy to Clipboard Toggle word wrap
   Note

   This causes a short period of downtime for the undercloud. The overcloud is still functional during the undercloud upgrade.

6. The default Provisioning/Control Plane network has changed from 192.0.2.0/24 to 192.168.24.0/24. If you used default network values in your previous undercloud.conf file, your Provisioning/Control Plane network is set to 192.0.2.0/24. This means you need to set certain parameters in your undercloud.conf file to continue using the 192.0.2.0/24 network. These parameters are:

   • local_ip
   • network_gateway
   • undercloud_public_vip
   • undercloud_admin_vip
   • network_cidr
   • masquerade_network
   • dhcp_start
   • dhcp_end

   Set the network values in undercloud.conf to ensure continued use of the 192.0.2.0/24 CIDR during future upgrades. Ensure your network configuration set correctly before running the openstack undercloud upgrade command.

7. Run yum to upgrade the director’s main packages:

   $ sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient

   Copy to Clipboard Toggle word wrap

8. Run the following command to upgrade the undercloud:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
9. Wait until the undercloud upgrade process completes.

Procedure

1. Log in to director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-11-rpms

   Copy to Clipboard Toggle word wrap

3. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-12-rpms

   Copy to Clipboard Toggle word wrap

4. Disable updates to the overcloud base images:

   $ sudo yum-config-manager --setopt=exclude=rhosp-director-images* --save

   Copy to Clipboard Toggle word wrap

5. Run yum to upgrade the director’s main packages:

   $ sudo yum update -y python-tripleoclient

   Copy to Clipboard Toggle word wrap
6. Edit the /home/stack/undercloud.conf file and check that the enabled_drivers parameter does not contain the pxe_ssh driver. This driver is deprecated in favor of the Virtual Baseboard Management Controller (VBMC) and removed from Red Hat OpenStack Platform. For more information about this new driver and migration instructions, see the Appendix "Virtual Baseboard Management Controller (VBMC)" in the Director Installation and Usage Guide.

7. Run the following command to upgrade the undercloud:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
8. Wait until the undercloud upgrade process completes.

Procedure

1. Log in to director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-12-rpms

   Copy to Clipboard Toggle word wrap

3. Set the RHEL version to RHEL 7.9:

   $ sudo subscription-manager release --set=7.9

   Copy to Clipboard Toggle word wrap

4. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-13-rpms

   Copy to Clipboard Toggle word wrap

5. Re-enable updates to the overcloud base images:

   $ sudo yum-config-manager --setopt=exclude= --save

   Copy to Clipboard Toggle word wrap

6. Run yum to upgrade the director’s main packages:

   $ sudo yum update -y python-tripleoclient

   Copy to Clipboard Toggle word wrap

7. Run the following command to upgrade the undercloud:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
8. Wait until the undercloud upgrade process completes.

9. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
10. Wait until the node boots.

Procedure

1. Log in to the undercloud as the stack user.

2. Stop and disable the openstack-glance-registry service:

   $ sudo systemctl stop openstack-glance-registry
   $ sudo systemctl disable openstack-glance-registry

   Copy to Clipboard Toggle word wrap

3. Stop and disable the mongod service:

   $ sudo systemctl stop mongod
   $ sudo systemctl disable mongod

   Copy to Clipboard Toggle word wrap

Procedure

1. To pull the images directly from registry.redhat.io in your overcloud deployment, an environment file is required to specify the image parameters. Run the following command to generate the container image environment file:

   (undercloud) $ sudo openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

2. Modify the overcloud_images.yaml file and include the following parameters to authenticate with registry.redhat.io during deployment:

   ContainerImageRegistryLogin: true
   ContainerImageRegistryCredentials:
     registry.redhat.io:
       <USERNAME>: <PASSWORD>

   Copy to Clipboard Toggle word wrap

   • Replace <USERNAME> and <PASSWORD> with your credentials for registry.redhat.io.

     The overcloud_images.yaml file contains the image locations on the undercloud. Include this file with your deployment.

     Note

     Before you run the openstack overcloud deploy command, you must log in to the remote registry:

     (undercloud) $ sudo docker login registry.redhat.io

     Copy to Clipboard Toggle word wrap

Procedure

1. Find the address of the local undercloud registry. The address uses the following pattern:

   <REGISTRY_IP_ADDRESS>:8787

   Copy to Clipboard Toggle word wrap

   Use the IP address of your undercloud, which you previously set with the local_ip parameter in your undercloud.conf file. For the commands below, the address is assumed to be 192.168.24.1:8787.

2. Log in to registry.redhat.io:

   (undercloud) $ docker login registry.redhat.io --username $RH_USER --password $RH_PASSWD

   Copy to Clipboard Toggle word wrap

3. Create a template to upload the images to the local registry, and the environment file to refer to those images:

   (undercloud) $ openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --push-destination=192.168.24.1:8787 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml \
     --output-images-file /home/stack/local_registry_images.yaml

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

4. Verify that the following two files have been created:

   • local_registry_images.yaml, which contains container image information from the remote source. Use this file to pull the images from the Red Hat Container Registry (registry.redhat.io) to the undercloud.
   • overcloud_images.yaml, which contains the eventual image locations on the undercloud. You include this file with your deployment.

5. Pull the container images from the remote registry and push them to the undercloud registry:

   (undercloud) $ openstack overcloud container image upload \
     --config-file  /home/stack/local_registry_images.yaml \
     --verbose

   Copy to Clipboard Toggle word wrap

   Pulling the required images might take some time depending on the speed of your network and your undercloud disk.

   Note

   The container images consume approximately 10 GB of disk space.

6. The images are now stored on the undercloud’s docker-distribution registry. To view the list of images on the undercloud’s docker-distribution registry, run the following command:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog | jq .repositories[]

   Copy to Clipboard Toggle word wrap
   Note

   The _catalog resource by itself displays only 100 images. To display more images, use the ?n=<interger> query string with the _catalog resource to display a larger number of images:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog?n=150 | jq .repositories[]

   Copy to Clipboard Toggle word wrap

   To view a list of tags for a specific image, use the skopeo command:

   (undercloud) $ curl -s http://192.168.24.1:8787/v2/rhosp13/openstack-keystone/tags/list | jq .tags

   Copy to Clipboard Toggle word wrap

   To verify a tagged image, use the skopeo command:

   (undercloud) $ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp13/openstack-keystone:13.0-44

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a template to pull images to the local registry:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud container image prepare \
     --namespace=rhosp13 \
     --prefix=openstack- \
     --output-images-file /home/stack/satellite_images

   Copy to Clipboard Toggle word wrap
   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.
   Note

   This version of the openstack overcloud container image prepare command targets the registry on the registry.redhat.io to generate an image list. It uses different values than the openstack overcloud container image prepare command used in a later step.

2. This creates a file called satellite_images with your container image information. You will use this file to synchronize container images to your Satellite 6 server.

3. Remove the YAML-specific information from the satellite_images file and convert it into a flat file containing only the list of images. The following sed commands accomplish this:

   (undercloud) $ awk -F ':' '{if (NR!=1) {gsub("[[:space:]]", ""); print $2}}' ~/satellite_images > ~/satellite_images_names

   Copy to Clipboard Toggle word wrap

   This provides a list of images that you pull into the Satellite server.

4. Copy the satellite_images_names file to a system that contains the Satellite 6 hammer tool. Alternatively, use the instructions in the Hammer CLI Guide to install the hammer tool to the undercloud.

5. Run the following hammer command to create a new product (OSP13 Containers) to your Satellite organization:

   $ hammer product create \
     --organization "ACME" \
     --name "OSP13 Containers"

   Copy to Clipboard Toggle word wrap

   This custom product will contain our images.

6. Add the base container image to the product:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhosp13/openstack-base \
     --name base

   Copy to Clipboard Toggle word wrap

7. Add the overcloud container images from the satellite_images file.

   $ while read IMAGE; do \
     IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \
     hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name $IMAGE \
     --name $IMAGENAME ; done < satellite_images_names

   Copy to Clipboard Toggle word wrap

8. Synchronize the container images:

   $ hammer product synchronize \
     --organization "ACME" \
     --name "OSP13 Containers"

   Copy to Clipboard Toggle word wrap

   Wait for the Satellite server to complete synchronization.

   Note

   Depending on your configuration, hammer might ask for your Satellite server username and password. You can configure hammer to automatically login using a configuration file. See the "Authentication" section in the Hammer CLI Guide.

9. If your Satellite 6 server uses content views, create a new content view version to incorporate the images.

10. Check the tags available for the base image:

    $ hammer docker tag list --repository "base" \
      --organization "ACME" \
      --product "OSP13 Containers"

    Copy to Clipboard Toggle word wrap

    This displays tags for the OpenStack Platform container images.

11. Return to the undercloud and generate an environment file for the images on your Satellite server. The following is an example command for generating the environment file:

    (undercloud) $ openstack overcloud container image prepare \
      --namespace=satellite6.example.com:5000 \
      --prefix=acme-osp13_containers- \
      --tag-from-label {version}-{release} \
      --output-env-file=/home/stack/templates/overcloud_images.yaml

    Copy to Clipboard Toggle word wrap
    Note

    This version of the openstack overcloud container image prepare command targets the Satellite server. It uses different values than the openstack overcloud container image prepare command used in a previous step.

    When running this command, include the following data:

    • --namespace - The URL and port of the registry on the Satellite server. The registry port on Red Hat Satellite is 5000. For example, --namespace=satellite6.example.com:5000.

      Note

      If you are using Red Hat Satellite version 6.10, you do not need to specify a port. The default port of 443 is used. For more information, see "How can we adapt RHOSP13 deployment to Red Hat Satellite 6.10?".

    • --prefix= - The prefix is based on a Satellite 6 convention for labels, which uses lower case characters and substitutes spaces for underscores. The prefix differs depending on whether you use content views:

      • If you use content views, the structure is [org]-[environment]-[content view]-[product]-. For example: acme-production-myosp13-osp13_containers-.
      • If you do not use content views, the structure is [org]-[product]-. For example: acme-osp13_containers-.
    • --tag-from-label {version}-{release} - Identifies the latest tag for each image.
    • -e - Include any environment files for optional services.
    • -r - Include a custom roles file.

    • --set ceph_namespace, --set ceph_image, --set ceph_tag - If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location. Note that ceph_image now includes a Satellite-specific prefix. This prefix is the same value as the --prefix option. For example:

      --set ceph_image=acme-osp13_containers-rhceph-3-rhel7

      Copy to Clipboard Toggle word wrap

      This ensures the overcloud uses the Ceph container image using the Satellite naming convention.

12. The overcloud_images.yaml file contains the image locations on the Satellite server. Include this file with your deployment.

Procedure

1. View your custom environment file:

   $ cat ~/templates/custom_environment.yaml

   Copy to Clipboard Toggle word wrap
2. Check for a resource_registry section in the file contents.

3. Check for any composable services in the resource_registry section. Composable services being with the following namespace:

   OS::TripleO::Services

   Copy to Clipboard Toggle word wrap

   For example, the following composable service is for the OpenStack Bare Metal Service (ironic) API:

   OS::TripleO::Services::IronicApi

   Copy to Clipboard Toggle word wrap

4. Check if the composable service maps to a Puppet-specific Heat template. For example:

   resource_registry:
     OS::TripleO::Services::IronicApi: /usr/share/openstack-triple-heat-template/puppet/services/ironic-api.yaml

   Copy to Clipboard Toggle word wrap

5. Check if a containerized version of the Heat template exists in /usr/share/openstack-triple-heat-template/docker/services/ and remap the service to the containerized version:

   resource_registry:
     OS::TripleO::Services::IronicApi: /usr/share/openstack-triple-heat-template/docker/services/ironic-api.yaml

   Copy to Clipboard Toggle word wrap

   Alternatively, use the updated environment files for the service, which are located in /usr/share/openstack-tripleo-heat-templates/environments/. For example, the latest environment file for enabling the OpenStack Bare Metal Service (ironic) is /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml, which contains the containerized service mappings.

   If the custom service does not use a containerised service, keep the mapping to the Puppet-specific Heat template.

Procedure

1. If you are using a director-managed or external Ceph Storage cluster, install the ceph-ansible package:

   1. Enable the Ceph Tools repository on the undercloud:

      [stack@director ~]$ sudo subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms

      Copy to Clipboard Toggle word wrap

   2. Install the ceph-ansible package to the undercloud:

      [stack@director ~]$ sudo yum install -y ceph-ansible

      Copy to Clipboard Toggle word wrap

2. Check your Ceph-specific environment files and ensure your Ceph-specific heat resources use containerized services:

   • For director-managed Ceph Storage clusters, ensure that the resources in the resource_register point to the templates in docker/services/ceph-ansible:

     resource_registry:
       OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mgr.yaml
       OS::TripleO::Services::CephMon: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mon.yaml
       OS::TripleO::Services::CephOSD: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-osd.yaml
       OS::TripleO::Services::CephClient: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-client.yaml

     Copy to Clipboard Toggle word wrap
     Important

     This configuration is included in the /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml environment file, which you can include with all future deployment commands with -e.

     Note

     If the environment or template file that you want to use in an environment is not present in the /usr/share directory, you must include the absolute path to the file.

   • For external Ceph Storage clusters, make sure the resource in the resource_register points to the template in docker/services/ceph-ansible:

     resource_registry:
       OS::TripleO::Services::CephExternal: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-external.yaml

     Copy to Clipboard Toggle word wrap
     Important

     This configuration is included in the /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml environment file, which you can include with all future deployment commands with -e.

3. For director-managed Ceph Storage clusters, use the new CephAnsibleDisksConfig parameter to define how your disks are mapped. Previous versions of Red Hat OpenStack Platform used the ceph::profile::params::osds hieradata to define the OSD layout. Convert this hieradata to the structure of the CephAnsibleDisksConfig parameter. The following examples show how to convert the hieradata to the structure of the CephAnsibleDisksConfig parameter in the case of collocated and non-collocated Ceph journal disks.

   Important

   You must set the osd_scenario. If you leave osd_scenario unset, it can result in a failed deployment.

   • In a scenario where the Ceph journal disks are collocated, if your hieradata contains the following:

     parameter_defaults:
       ExtraConfig:
         ceph::profile::params::osd_journal_size: 512
         ceph::profile::params::osds:
           '/dev/sdb': {}
           '/dev/sdc': {}
           '/dev/sdd': {}

     Copy to Clipboard Toggle word wrap

     Convert the hieradata in the following way with the CephAnsibleDisksConfig parameter and set ceph::profile::params::osds to {}:

     parameter_defaults:
       CephAnsibleDisksConfig:
         devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
         journal_size: 512
         osd_scenario: collocated
       ExtraConfig:
           ceph::profile::params::osds: {}

     Copy to Clipboard Toggle word wrap

   • In a scenario where the journals are on faster dedicated devices and are non-collocated, if the hieradata contains the following:

     parameter_defaults:
       ExtraConfig:
         ceph::profile::params::osd_journal_size: 512
         ceph::profile::params::osds:
           '/dev/sdb':
              journal: ‘/dev/sdn’
           '/dev/sdc':
              journal: ‘/dev/sdn’
           '/dev/sdd':
              journal: ‘/dev/sdn’

     Copy to Clipboard Toggle word wrap

     Convert the hieradata in the following way with the CephAnsibleDisksConfig parameter and set ceph::profile::params::osds to {}:

     parameter_defaults:
       CephAnsibleDisksConfig:
         devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
         dedicated_devices:
         - /dev/sdn
         - /dev/sdn
         - /dev/sdn
         journal_size: 512
         osd_scenario: non-collocated
       ExtraConfig:
         ceph::profile::params::osds: {}

     Copy to Clipboard Toggle word wrap

   For a full list of OSD disk layout options used in ceph-ansible, view the sample file in /usr/share/ceph-ansible/group_vars/osds.yml.sample.

4. Include the new Ceph configuration environment files with future deployment commands using the -e option. This includes the following files:

   • Director-managed Ceph Storage:

     • /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml.
     • The environment file with the Ansible-based disk-mapping.
     • Any additional environment files with Ceph Storage customization.

   • External Ceph Storage:

     • /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml
     • Any additional environment files with Ceph Storage customization.

Procedure

1. The director’s dynamic Ansible script has updated to the OpenStack Platform 12 version, which uses the RoleNetHostnameMap Heat parameter in the overcloud plan to define the inventory. However, the overcloud currently uses the OpenStack Platform 11 template versions, which do not have the RoleNetHostnameMap parameter. This means you need to create a temporary static inventory file, which you can generate with the following command:

   $ openstack server list -c Networks -f value | cut -d"=" -f2 > overcloud_hosts

   Copy to Clipboard Toggle word wrap

2. Create an Ansible playbook (undercloud-ca.yml) that contains the following:

   ---
   - name: Add undercloud CA to overcloud nodes
     hosts: all
     user: heat-admin
     become: true
     vars:
       ca_certificate: /etc/pki/ca-trust/source/anchors/cm-local-ca.pem
     tasks:
       - name: Copy undercloud CA
         copy:
           src: "{{ ca_certificate }}"
           dest: /etc/pki/ca-trust/source/anchors/
       - name: Update trust
         command: "update-ca-trust extract"
       - name: Get the swift endpoint
         shell: |
           sudo hiera swift::keystone::auth::public_url | awk -F/ '{print $3}'
         register: swift_endpoint
         delegate_to: 127.0.0.1
         become: yes
         become_user: stack
       - name: Verify URL
         uri:
           url: https://{{ swift_endpoint.stdout }}/healthcheck
           return_content: yes
         register: verify
       - name: Report output
         debug:
           msg: "{{ ansible_hostname }} can access the undercloud's Public API"
         when: verify.content == "OK"

   Copy to Clipboard Toggle word wrap

   This playbook contains multiple tasks that perform the following on each node:

   • Copy the undercloud’s certificate authority file to the overcloud node. If generated by the undercloud, the default location is /etc/pki/ca-trust/source/anchors/cm-local-ca.pem.
   • Execute the command to update the certificate authority trust database on the overcloud node.
   • Checks the undercloud’s Object Storage Public API from the overcloud node and reports if successful.

3. Run the playbook with the following command:

   $ ansible-playbook -i overcloud_hosts undercloud-ca.yml

   Copy to Clipboard Toggle word wrap

   This uses the temporary inventory to provide Ansible with your overcloud nodes.

   If using a custom certificate authority file, you can change the ca_certificate variable to a location. For example:

   $ ansible-playbook -i overcloud_hosts undercloud-ca.yml -e ca_certificate=/home/stack/ssl/ca.crt.pem

   Copy to Clipboard Toggle word wrap

4. The resulting Ansible output should show a debug message for node. For example:

   ok: [192.168.24.100] => {
       "msg": "overcloud-controller-0 can access the undercloud's Public API"
   }

   Copy to Clipboard Toggle word wrap

Procedure

1. Select an environment file and the check if it has an ExtraConfig parameter:

   $ grep ExtraConfig ~/templates/custom-config.yaml

   Copy to Clipboard Toggle word wrap
2. If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig) in the chosen file, check the full parameter structure in that file.

3. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value, it might have been been replaced with a parameter with an actual Puppet class. For example:

   parameter_defaults:
     ExtraConfig:
       neutron::config::dhcp_agent_config:
         'DEFAULT/dnsmasq_local_resolv':
           value: 'true'

   Copy to Clipboard Toggle word wrap

4. Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:

   $ grep dnsmasq_local_resolv

   Copy to Clipboard Toggle word wrap

   If so, change to the new interface.

5. The following are examples to demonstrate the change in syntax:

   • Example 1:

     parameter_defaults:
       ExtraConfig:
         neutron::config::dhcp_agent_config:
           'DEFAULT/dnsmasq_local_resolv':
             value: 'true'

     Copy to Clipboard Toggle word wrap

     Changes to:

     parameter_defaults:
       ExtraConfig:
         neutron::agents::dhcp::dnsmasq_local_resolv: true

     Copy to Clipboard Toggle word wrap

   • Example 2:

     parameter_defaults:
       ExtraConfig:
         ceilometer::config::ceilometer_config:
           'oslo_messaging_rabbit/rabbit_qos_prefetch_count':
             value: '32'

     Copy to Clipboard Toggle word wrap

     Changes to:

     parameter_defaults:
       ExtraConfig:
         oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'

     Copy to Clipboard Toggle word wrap