Chapter 11. Performing a minor update of the RHOSP overcloud with director Operator

Procedure

1. Change to the openstack project:

   $ oc project openstack

2. Edit the CSV file:

   $ oc edit csv

3. Update the following values to the new RHOSP minor version:

   OPENSTACKCLIENT_IMAGE_URL_DEFAULT
   HEAT_API_IMAGE_URL_DEFAULT
   HEAT_ENGINE_IMAGE_URL_DEFAULT
   MARIADB_IMAGE_URL_DEFAULT
   RABBITMQ_IMAGE_URL_DEFAULT

4. Delete any existing ephemeral Heat instances:

   $ oc delete openstackephemeralheat --all

5. Remove the current imageURL from the openstackclient custom resource to update the pod to the new image:

   $ oc patch openstackclient -n openstack openstackclient --type=json -p="[{'op': 'remove', 'path': '/spec/imageURL'}]"

Procedure

1. Copy the overcloud subscription management environment file, rhsm.yaml, to openstackclient:

   $ oc cp rhsm.yaml openstackclient:/home/cloud-admin/rhsm.yaml

2. Access the remote shell for the openstackclient pod:

   $ oc rsh openstackclient

3. Open the rhsm.yaml file and check if your subscription management configuration includes the rhsm_release parameter. If the rhsm_release parameter is not present, add it and set it to 9.2:

   parameter_defaults:
     RhsmVars:
       …​
       rhsm_username: "myusername"
       rhsm_password: "p@55w0rd!"
       rhsm_org_id: "1234567"
       rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
       rhsm_method: "portal"
       rhsm_release: "9.2"

4. Save the rhsm.yaml file.

5. Create a playbook named set_release.yaml that contains a task to lock the operating system version to RHEL 9.2 on all nodes:

   - hosts: all
     gather_facts: false
     tasks:
       - name: set release to 9.2
         command: subscription-manager release --set=9.2
         become: true

6. Run the set_release.yaml playbook on the openstackclient pod:

   $ ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/set_release.yaml --limit Controller,Compute

   Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Red Hat Ceph Storage nodes because you might have a different subscription for these nodes.

   Note

   To manually lock a node to a version, log in to the node and run the subscription-manager release command:

   $ sudo subscription-manager release --set=9.2

7. Exit the remote shell for the openstackclient pod:

   $ exit

Procedure

1. Open the rhsm.yaml file and update the rhsm_repos parameter to the correct repository versions:

   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-9-for-x86_64-baseos-e4s-rpms
         - rhel-9-for-x86_64-appstream-e4s-rpms
         - rhel-9-for-x86_64-highavailability-e4s-rpms
         - openstack-17.1-for-rhel-9-x86_64-rpms
         - fast-datapath-for-rhel-9-x86_64-rpms

2. Save the rhsm.yaml file.

3. Access the remote shell for the openstackclient pod:

   $ oc rsh openstackclient

4. Create a playbook named update_rhosp_repos.yaml that contains a task to set the repositories to RHOSP 17.1 on all nodes:

   - hosts: all
     gather_facts: false
     tasks:
       - name: change osp repos
         command: subscription-manager repos --enable=openstack-17.1-for-rhel-9-x86_64-rpms
         become: true

5. Run the update_rhosp_repos.yaml playbook on the openstackclient pod:

   $ ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_rhosp_repos.yaml --limit Controller,Compute

   Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Red Hat Ceph Storage nodes because they use a different subscription.

6. Create a playbook named update_ceph_repos.yaml that contains a task to set the repositories to RHOSP 17.1 on all Red Hat Ceph Storage nodes:

   - hosts: all
     gather_facts: false
     tasks:
       - name: change ceph repos
         command: subscription-manager repos --enable=openstack-17.1-deployment-tools-for-rhel-9-x86_64-rpms
         become: true

7. Run the update_ceph_repos.yaml playbook on the openstackclient pod:

   $ ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory /home/cloud-admin/update_ceph_repos.yaml --limit CephStorage

   Use the --limit option to apply the content to Red Hat Ceph Storage nodes.

8. Exit the remote shell for the openstackclient pod:

   $ exit

Procedure

1. Edit the container preparation file. The default name for this file is containers-prepare-parameter.yaml.

2. Ensure the tag parameter is set to 17.1 for each rule set:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: false
       set:
         ...
         tag: '17.1'
       tag_from_label: '{version}-{release}'

   Note

   If you do not want to use a specific tag for the update, such as 17.1 or 17.1.1, remove the tag key-value pair and specify tag_from_label only. The tag_from_label tag uses the installed Red Hat OpenStack Platform (RHOSP) version to determine the value for the tag to use as part of the update process. For more information about version tagging, see Guidelines for container image tagging in Customizing your Red Hat OpenStack Platform deployment.

3. Save the containers-prepare-parameter.yaml file.

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh openstackclient

2. Log in to a Controller node and run the Pacemaker command to disable fencing:

   $ ssh <controller-0.ctlplane> "sudo pcs property set stonith-enabled=false"

   • Replace <controller-0.ctlplane> with the name of your Controller node.

3. Exit the remote shell for the openstackclient pod:

   $ exit

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

Verification

1. Verify that the services are enabled.

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check that all services are enabled:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check that all containers on the node are active:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo cephadm shell -- ceph osd set noout
   $ sudo cephadm shell -- ceph osd set norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you set the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring.

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the Ceph cluster status:

   $ sudo cephadm shell -- ceph status

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes.

7. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and enable Ceph cluster rebalancing:

   $ sudo cephadm shell -- ceph osd unset noout
   $ sudo cephadm shell -- ceph osd unset norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you unset the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo cephadm shell ceph status

Migrating instances workflow

1. Decide whether to migrate instances to another Compute node before rebooting the node.
2. Select and disable the Compute node that you want to reboot so that it does not provision new instances.
3. Migrate the instances to another Compute node.
4. Reboot the empty Compute node.
5. Enable the empty Compute node.

Procedure

1. Log in to the undercloud as the stack user.

2. Retrieve a list of your Compute nodes to identify the host name of the node that you want to reboot:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack compute service list

   Identify the host name of the Compute node that you want to reboot.

3. Disable the Compute service on the Compute node that you want to reboot:

   (overcloud)$ openstack compute service list
   (overcloud)$ openstack compute service set <hostname> nova-compute --disable

   • Replace <hostname> with the host name of your Compute node.

4. List all instances on the Compute node:

   (overcloud)$ openstack server list --host <hostname> --all-projects

5. Optional: To migrate the instances to another Compute node, complete the following steps:

   1. If you decide to migrate the instances to another Compute node, use one of the following commands:

      • To migrate the instance to a different host, run the following command:

        (overcloud) $ openstack server migrate <instance_id> --live <target_host> --wait

        • Replace <instance_id> with your instance ID.
        • Replace <target_host> with the host that you are migrating the instance to.

      • Let nova-scheduler automatically select the target host:

        (overcloud) $ nova live-migration <instance_id>

      • Live migrate all instances at once:

        $ nova host-evacuate-live <hostname>

        Note

        The nova command might cause some deprecation warnings, which are safe to ignore.

   2. Wait until migration completes.

   3. Confirm that the migration was successful:

      (overcloud) $ openstack server list --host <hostname> --all-projects

   4. Continue to migrate instances until none remain on the Compute node.

6. Log in to the Compute node and reboot the node:

   [tripleo-admin@overcloud-compute-0 ~]$ sudo reboot

7. Wait until the node boots.

8. Re-enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname>  nova-compute --enable

9. Check that the Compute node is enabled:

   (overcloud) $ openstack compute service list

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the validation:

   $ validation run -i ~/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml --group post-update

   • Replace <stack> with the name of the stack.

Verification

1. To view the results of the validation report, see Viewing validation history in Installing and managing Red Hat OpenStack Platform with director.