Performing a minor update of Red Hat OpenStack Platform

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.

4. Check if your subscription management configuration includes the rhsm_release parameter. If the rhsm_release parameter is not present, add it and set it to 9.2:

   parameter_defaults:
     RhsmVars:
       …​
       rhsm_username: "myusername"
       rhsm_password: "p@55w0rd!"
       rhsm_org_id: "1234567"
       rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
       rhsm_method: "portal"
       rhsm_release: "9.2"

5. Save the overcloud subscription management environment file.

6. Create a playbook that contains a task to lock the operating system version to RHEL 9.2 on all nodes:

   $ cat > ~/set_release.yaml <<'EOF'
   - hosts: all
     gather_facts: false
     tasks:
       - name: set release to 9.2
         command: subscription-manager release --set=9.2
         become: true
   EOF

7. Run the set_release.yaml playbook:

   $ ansible-playbook -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml -f 25 ~/set_release.yaml --limit <undercloud>, <Controller>, <Compute>

   • Replace <stack> with the name of your stack.
   • Use the --limit option to apply the content to all RHOSP nodes. Replace <undercloud>, <Controller>, <Compute> with the Ansible groups in your environment that contain those nodes. Do not run this playbook against Ceph Storage nodes because you might have a different subscription for these nodes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.

4. Check the rhsm_repos parameter in your subscription management configuration. If the rhsm_repos parameter is using the RHOSP 17.1 repositories, change the repository to the correct versions:

   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-9-for-x86_64-baseos-e4s-rpms
         - rhel-9-for-x86_64-appstream-e4s-rpms
         - rhel-9-for-x86_64-highavailability-e4s-rpms
         - openstack-17.1-for-rhel-9-x86_64-rpms
         - fast-datapath-for-rhel-9-x86_64-rpms

5. Save the overcloud subscription management environment file.

6. Create a playbook that contains a task to set the repositories to RHOSP 17.1 on all nodes:

   $ cat > ~/update_rhosp_repos.yaml <<'EOF'
   - hosts: all
     gather_facts: false
     tasks:
       - name: change osp repos
         command: subscription-manager repos --enable=openstack-17.1-for-rhel-9-x86_64-rpms
         become: true
   EOF

7. Run the update_rhosp_repos.yaml playbook:

   $ ansible-playbook -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml -f 25 ~/update_rhosp_repos.yaml --limit <undercloud>,<Controller>,<Compute>

   • Replace <stack> with the name of your stack.
   • Use the --limit option to apply the content to all RHOSP nodes. Replace <undercloud>, <Controller>, and <Compute> with the Ansible groups in your environment that contain those nodes. Do not run this playbook against Ceph Storage nodes because they usually use a different subscription.

8. Create a playbook that contains a task to set the repositories to RHOSP 17.1 on all ceph storage nodes:

   $ cat > ~/update_ceph_repos.yaml <<'EOF'
   - hosts: all
     gather_facts: false
     tasks:
       - name: change ceph repos
         command: subscription-manager repos --enable=openstack-17.1-deployment-tools-for-rhel-9-x86_64-rpms
         become: true
   EOF

9. Run the update_ceph_repos.yaml playbook:

   $ ansible-playbook -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml -f 25 ~/update_ceph_repos.yaml --limit CephStorage

   Use the --limit option to apply the content to Ceph Storage nodes.

Procedure

1. Edit the container preparation file. The default name for this file is usually containers-prepare-parameter.yaml.

2. Ensure that the tag parameter is set to 17.1 for each rule set:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ...
         tag: '17.1'
       tag_from_label: '{version}-{release}'

   Note

   If you do not want to use a specific tag for the update, such as 17.1 or 17.1.1, remove the tag key-value pair and specify tag_from_label only. This uses the installed Red Hat OpenStack Platform version to determine the value for the tag to use as part of the update process. For more information about version tagging, see Guidelines for container image tagging in Customizing your Red Hat OpenStack Platform deployment.

3. Save this file.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. For each Controller node, log in to the Controller node and run the Pacemaker command to disable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes at /etc/hosts or /var/lib/mistral.

4. If you use SBD fencing, disable SBD fencing on the pacemaker_remote nodes:

   # pcs property set stonith-watchdog-timeout=0 --force

   Note

   Ensure that you take note of the original value of the watchdog timer device interval. You must reset the watchdog timer device interval to its original value after you upgrade the control plane nodes. For more information, see Re-enabling fencing in the overcloud.

5. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the update process.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Install the packages for validation:

   $ sudo dnf -y update openstack-tripleo-validations python3-validations-libs validations-common

4. Run the validation:

   $ validation run -i ~/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml --group pre-update

   • Replace <stack> with the name of the stack.

Verification

1. To view the results of the validation report, see Viewing validation history in Installing and managing Red Hat OpenStack Platform with director.

Procedure

1. Validate your SSH key size:

   ssh-keygen -l -f ~/.ssh/id_rsa.pub

   Example output:

   1024 SHA256:Xqz0Xz0/aJua6B3qRD7VsLr6n/V3zhmnGSkcFR6FlJw stack@director.example.local (RSA)

2. If your SSH key is less than 2048 bits, you must rotate out the SSH key before continuing. For more information, see Updating SSH keys in your OpenStack environment in Hardening Red Hat OpenStack Platform.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Update the director main packages with the dnf update command:

   $ sudo dnf update -y python3-tripleoclient ansible-*

4. Update the undercloud environment:

   $ openstack undercloud upgrade

5. Wait until the undercloud update process completes.

6. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

7. Wait until the node boots.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

4. Extract the archives:

   cd ~/images
   for i in /usr/share/rhosp-director-images/ironic-python-agent-latest-17.1.tar /usr/share/rhosp-director-images/overcloud-hardened-uefi-full-latest-17.1.tar; do tar -xvf $i; done
   cd ~

5. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

6. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

7. Verify the existence of the new images:

   $ ls -l /var/lib/ironic/httpboot /var/lib/ironic/images

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update preparation command:

   $ openstack overcloud update prepare \
       --templates \
       --stack <stack_name> \
       -r <roles_data_file> \
       -n <network_data_file> \
       -e <environment_file> \
       -e <environment_file> \
       ...

   Include the following options relevant to your environment:

   • Include the /usr/share/openstack-tripleo-heat-templates/environments/rhsm.yaml file with the -e option to apply the RHOSP 17.1 repositories.
   • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <stack_name> with the name of your stack.
   • If you use your own custom roles, use the -r option to include the custom roles (<roles_data_file>) file.
   • If you use custom networks, use the -n option to include your composable network in the (<network_data_file>) file.
   • If you deploy a high availability cluster, include the --ntp-server option in the update preparation command, or include the NtpServer parameter and value in your environment file.
   • Include any custom configuration environment files with the -e option.
4. Wait until the update preparation process completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the openstack overcloud external-update run command against tasks that have the container_image_prepare tag:

   $ openstack overcloud external-update run --stack <stack_name> --tags container_image_prepare

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the openstack overcloud external-update run command against the tasks that have the ovn tag:

   $ openstack overcloud external-update run --stack <stack_name> --tags ovn

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the ovn-controller container update completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the openstack overcloud external-update run command with the ha_image_update tag:

   $ openstack overcloud external-update run --stack <stack_name> --tags ha_image_update

   • If the name of your undercloud stack is different from the default stack name undercloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack <stack_name> --limit Controller

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the Controller node update completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack <stack_name> --limit <non_pcs_role_0>
   $ openstack overcloud update run --stack <stack_name> --limit <non_pcs_role_1>
   $ openstack overcloud update run --stack <stack_name> --limit <non_pcs_role_2>

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
   • Replace <non_pcs_role_0>, <non_pcs_role_1>, and <non_pcs_role_2> with the names of your composable roles with non-Pacemaker services.
4. Wait until the update completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack <stack_name> --limit Compute

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the Compute node update completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack <stack_name> --limit ComputeHCI

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the node update completes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack <stack_name> --limit DistributedComputeHCI

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the DistributedComputeHCI node update completes.
5. Use the same process to update DistributedComputeHCIScaleOut nodes.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the update command:

   $ openstack overcloud update run --stack  <stack_name> --limit CephStorage

   • If the name of your overcloud stack is different from the default stack name overcloud, set your stack name with the --stack option and replace <stack_name> with the name of your stack.
4. Wait until the node update completes.

Procedure

1. Log in to a Controller node.

2. Check the health of the cluster:

   $ sudo cephadm shell -- ceph health

   Note

   If the Ceph Storage cluster is healthy, the command returns a result of HEALTH_OK. If the command returns a different result, review the status of the cluster and contact Red Hat support before continuing the update. For more information, see Upgrade a Red Hat Ceph Storage cluster using cephadm in the Red Hat Ceph Storage Upgrade Guide or Upgrade a Red Hat Ceph Storage cluster using cephadm in the Red Hat Ceph Storage 6 Upgrade Guide.

3. Optional: Check which images should be included in the Ceph Storage cluster update:

   $ openstack tripleo container image list -f value |  awk -F '//' '/ceph/ {print $2}'

4. Update the cluster to the latest Red Hat Ceph Storage version:

   $ sudo cephadm shell -- ceph orch upgrade start --image <image_name>: <version>

   • Replace <image_name> with the name of the Ceph Storage cluster image.
   • Replace <version> with the target version to which you are updating the Ceph Storage cluster.

5. Wait until the Ceph Storage container update completes. To monitor the status of the update, run the following command:

    sudo cephadm shell -- ceph orch upgrade status

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the openstack overcloud external-update run command against tasks that use the online_upgrade tag:

   $ openstack overcloud external-update run --stack <stack_name> --tags online_upgrade

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to re-enable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command.

4. If you use SBD fencing, reset the watchdog timer device interval to its original value before you disabled it:

   # pcs property set stonith-watchdog-timeout=<interval>

   • Replace <interval> with the original value of the watchdog timer device, for example, 10.
5. In the fencing.yaml environment file, set the EnableFencing parameter to true.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

Verification

1. Verify that the services are enabled.

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check that all services are enabled:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check that all containers on the node are active:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo cephadm shell -- ceph osd set noout
   $ sudo cephadm shell -- ceph osd set norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you set the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring.

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the Ceph cluster status:

   $ sudo cephadm shell -- ceph status

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes.

7. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and enable Ceph cluster rebalancing:

   $ sudo cephadm shell -- ceph osd unset noout
   $ sudo cephadm shell -- ceph osd unset norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you unset the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo cephadm shell ceph status

Migrating instances workflow

1. Decide whether to migrate instances to another Compute node before rebooting the node.
2. Select and disable the Compute node that you want to reboot so that it does not provision new instances.
3. Migrate the instances to another Compute node.
4. Reboot the empty Compute node.
5. Enable the empty Compute node.

Procedure

1. Log in to the undercloud as the stack user.

2. Retrieve a list of your Compute nodes to identify the host name of the node that you want to reboot:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack compute service list

   Identify the host name of the Compute node that you want to reboot.

3. Disable the Compute service on the Compute node that you want to reboot:

   (overcloud)$ openstack compute service list
   (overcloud)$ openstack compute service set <hostname> nova-compute --disable

   • Replace <hostname> with the host name of your Compute node.

4. List all instances on the Compute node:

   (overcloud)$ openstack server list --host <hostname> --all-projects

5. Optional: To migrate the instances to another Compute node, complete the following steps:

   1. If you decide to migrate the instances to another Compute node, use one of the following commands:

      • To migrate the instance to a different host, run the following command:

        (overcloud) $ openstack server migrate <instance_id> --live <target_host> --wait

        • Replace <instance_id> with your instance ID.
        • Replace <target_host> with the host that you are migrating the instance to.

      • Let nova-scheduler automatically select the target host:

        (overcloud) $ nova live-migration <instance_id>

      • Live migrate all instances at once:

        $ nova host-evacuate-live <hostname>

        Note

        The nova command might cause some deprecation warnings, which are safe to ignore.

   2. Wait until migration completes.

   3. Confirm that the migration was successful:

      (overcloud) $ openstack server list --host <hostname> --all-projects

   4. Continue to migrate instances until none remain on the Compute node.

6. Log in to the Compute node and reboot the node:

   [tripleo-admin@overcloud-compute-0 ~]$ sudo reboot

   Note

   After a Compute node reboot, instances with VNC enabled will have a USB keyboard device automatically added to their configuration. If you experience console keyboard input issues for new instances after rebooting a Compute node, see Unable to access console of virtual machine after Compute node reboot. For existing instances, stop the instance, run the following command, and restart the instance:

   nova-manage image_property set --property hw_input_bus=virtio <instance_uuid>

7. Wait until the node boots.

8. Re-enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname>  nova-compute --enable

9. Check that the Compute node is enabled:

   (overcloud) $ openstack compute service list

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the validation:

   $ validation run -i ~/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml --group post-update

   • Replace <stack> with the name of the stack.

Verification

1. To view the results of the validation report, see Viewing validation history in Installing and managing Red Hat OpenStack Platform with director.