Keeping Red Hat OpenStack Platform Updated

Procedure

1. Log into the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Create a static inventory file of your overcloud:

   $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

   Copy to Clipboard Toggle word wrap

   If you use an overcloud name different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

4. Create a playbook that contains a task to lock the operating system version to Red Hat Enterprise Linux 8.1 on all nodes:

   $ cat > ~/set_release.yaml <<'EOF'
   - hosts: overcloud,undercloud
     gather_facts: false
     tasks:
       - name: set release to 8.1
         command: subscription-manager release --set=8.1
         become: true
   EOF

   Copy to Clipboard Toggle word wrap

5. Run the set_release.yaml playbook:

   $ ansible-playbook -i ~/inventory.yaml -f 25 ~/set_release.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap
3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.

4. Check the rhsm_repos parameter in your subscription management configuration. If this parameter does not include the EUS repositories, change the relevant repositories to the EUS versions:

   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-8-for-x86_64-baseos-eus-rpms
         - rhel-8-for-x86_64-appstream-eus-rpms
         - rhel-8-for-x86_64-highavailability-eus-rpms
         - ansible-2.9-for-rhel-8-x86_64-rpms
         - advanced-virt-for-rhel-8-x86_64-rpms
         - openstack-beta-for-rhel-8-x86_64-rpms
         - rhceph-4-osd-for-rhel-8-x86_64-rpms
         - rhceph-4-mon-for-rhel-8-x86_64-rpms
         - rhceph-4-tools-for-rhel-8-x86_64-rpms
         - fast-datapath-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap
5. Save the overcloud subscription management environment file.

6. Create a static inventory file of your overcloud:

   $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

   Copy to Clipboard Toggle word wrap

   If you use an overcloud name different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

7. Create a playbook that contains a task to set the repositories to Red Hat Enterprise Linux 8.1 EUS on all nodes:

   $ cat > ~/change_eus.yaml <<'EOF'
   - hosts: overcloud,undercloud
     gather_facts: false
     tasks:
       - name: change to eus repos
         command: subscription-manager repos --disable=rhel-8-for-x86_64-baseos-rpms --disable=rhel-8-for-x86_64-appstream-rpms --disable=rhel-8-for-x86_64-highavailability-rpms --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms
         become: true
   EOF

   Copy to Clipboard Toggle word wrap

8. Run the change_eus.yaml playbook:

   $ ansible-playbook -i ~/inventory.yaml -f 25 ~/change_eus.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the director as the stack user.

2. Run dnf to upgrade the director’s main packages:

   $ sudo dnf update -y python3-tripleoclient* openstack-tripleo-common openstack-tripleo-heat-templates tripleo-ansible

   Copy to Clipboard Toggle word wrap

3. The director uses the openstack undercloud upgrade command to update the undercloud environment. Run the command:

   $ openstack undercloud upgrade

   Copy to Clipboard Toggle word wrap
4. Wait until the undercloud upgrade process completes.

5. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
6. Wait until the node boots.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

   Copy to Clipboard Toggle word wrap

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.0.tar; do tar -xvf $i; done
   $ cd ~

   Copy to Clipboard Toggle word wrap

4. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

   Copy to Clipboard Toggle word wrap

5. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

   Copy to Clipboard Toggle word wrap

6. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /var/lib/ironic/httpboot

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the update preparation command:

   $ openstack overcloud update prepare \
       --templates \
       -r <ROLES DATA FILE> \
       -n <NETWORK DATA FILE> \
       -e <ENVIRONMENT FILE> \
       -e <ENVIRONMENT FILE> \
       --stack <STACK_NAME> \
       …​

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e)
   • If using your own custom roles, include your custom roles (roles_data) file (-r)
   • If using custom networks, include your composable network (network_data) file (-n)
   • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your stack.
3. Wait until the update preparation completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the openstack overcloud external-update run command against tasks tagged with the container_image_prepare tag:

   $ openstack overcloud external-update run --stack STACK_NAME --tags container_image_prepare

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the update command:

   $ openstack overcloud update run --stack STACK_NAME --limit Controller --playbook all

   Copy to Clipboard Toggle word wrap
3. Wait until the Controller node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the update command:

   $ openstack overcloud update run --stack STACK_NAME --limit Compute --playbook all

   Copy to Clipboard Toggle word wrap
3. Wait until the Compute node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the update command:

   $ openstack overcloud update run --stack _STACK_NAME_ --limit ComputeHCI --playbook all

   Copy to Clipboard Toggle word wrap
3. Wait until the node update completes.

4. Run the Ceph Storage update command. For example:

   $ openstack overcloud external-update run --stack _STACK_NAME_ --tags ceph

   Copy to Clipboard Toggle word wrap
5. Wait until the Compute HCI node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Update group nodes.

   To update all nodes in a group:

   $ openstack overcloud update run --nodes <GROUP_NAME>

   Copy to Clipboard Toggle word wrap

   To update a single node in a group:

   $ openstack overcloud update run --nodes <GROUP_NAME> [NODE_INDEX]

   Copy to Clipboard Toggle word wrap
   Note

   Ensure that you update all nodes if you choose to update nodes individually.

   The index of the first node in a group is zero (0). For example, to update the first node in a group named CephStorage, the command is:

   openstack overcloud update run --nodes CephStorage[0]

3. Wait until the node update completes.

4. Run the Ceph Storage container update command:

   $ openstack overcloud external-update run --stack _STACK_NAME_ --tags ceph

   Copy to Clipboard Toggle word wrap
5. Wait until the Ceph Storage container update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the openstack overcloud external-update run command against tasks tagged with the online_upgrade tag:

   $ openstack overcloud external-update run --stack STACK_NAME --tags online_upgrade

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the update finalization command:

   $ openstack overcloud update converge \
       --templates \
       -r <ROLES DATA FILE> \
       -n <NETWORK DATA FILE> \
       -e <ENVIRONMENT FILE> \
       -e <ENVIRONMENT FILE> \
       --stack <STACK_NAME> \
       …​

   Copy to Clipboard Toggle word wrap

   Include the following options relevant to your environment:

   + * Custom configuration environment files (-e) * If using your own custom roles, include your custom roles (roles_data) file (-r) * If using custom networks, include your composable network (network_data) file (-n) * If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your stack.

3. Wait until the update finalization completes.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

   Copy to Clipboard Toggle word wrap

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

      Copy to Clipboard Toggle word wrap

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

      Copy to Clipboard Toggle word wrap

   3. If the node uses containerized services, check that all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

      Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and re-enable cluster rebalancing:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

   Identify the UUID of the Compute node that you want to reboot.

3. From the undercloud, select a Compute node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

   Copy to Clipboard Toggle word wrap

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap
5. If you decide not to migrate instances, skip to this step.

6. If you decide to migrate the instances to another Compute node, use one of the following commands:

   • Migrate the instance to a different host:

     (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

     Copy to Clipboard Toggle word wrap

   • Let nova-scheduler automatically select the target host:

     (overcloud) $ nova live-migration [instance-id]

     Copy to Clipboard Toggle word wrap

   • Live migrate all instances at once:

     $ nova host-evacuate-live [hostname]

     Copy to Clipboard Toggle word wrap
     Note

     The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm that the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap
9. Continue to migrate instances until none remain on the chosen Compute node.

10. Log in to the Compute node and reboot the node:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

    Copy to Clipboard Toggle word wrap
11. Wait until the node boots.

12. Re-enable the Compute node:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

    Copy to Clipboard Toggle word wrap

13. Check that the Compute node is enabled:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Log in to the undercloud as the stack user.

3. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Copy to Clipboard Toggle word wrap

   Identify the UUID of the Compute node you aim to reboot.

4. From the undercloud, select a Compute node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

   Copy to Clipboard Toggle word wrap

5. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap

6. Use one of the following commands to migrate your instances:

   1. Migrate the instance to a specific host of your choice:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

      Copy to Clipboard Toggle word wrap

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

      Copy to Clipboard Toggle word wrap

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Copy to Clipboard Toggle word wrap
      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until the migration completes.

8. Confirm that the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

   Copy to Clipboard Toggle word wrap
9. Continue migrating instances until none remain on the chosen Compute node.

10. Log in to a Ceph MON or a Controller node and check the cluster status:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph -s

    Copy to Clipboard Toggle word wrap

    Check that the pgmap reports all pgs as normal (active+clean).

11. Reboot the Compute HCI node:

    $ sudo reboot

    Copy to Clipboard Toggle word wrap
12. Wait until the node boots.

13. Enable the Compute node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

    Copy to Clipboard Toggle word wrap

14. Verify that the Compute node is enabled:

    (overcloud) $ openstack compute service list

    Copy to Clipboard Toggle word wrap
15. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

16. When complete, log in to a Ceph MON or Controller node and enable cluster rebalancing again:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph osd unset noout
    $ sudo podman exec $CEPH_MON_CONTAINER ceph osd unset norebalance

    Copy to Clipboard Toggle word wrap

17. Perform a final status check to verify the cluster reports HEALTH_OK:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph status

    Copy to Clipboard Toggle word wrap