Chapter 3. Release information

Before this update, you could not suspend and resume instances with mediated (mdev) devices, such as vGPUs, because of a known limitation with the Compute service (nova) and the libvirt driver.

This limitation was addressed in the libvirt driver. Now you can suspend and resume instances with mdev devices in RHOSP 16.2, which runs on libvirt 6.0.0 in RHEL 8.4.

Before this update, the Block Storage API supported the creation of a Block Storage multi-attach volume by passing a parameter in the volume-create request, even though this method of creating a multi-attach volume was deprecated for removal. This method can lead to data loss when creating a multi-attach volume on a back end that does not support multi-attach volumes.

The openstack and cinder CLI only supported creating a multi-attach volume by using a multi-attach volume-type.

With this update, the Block Storage API also only supports creating a multi-attach volume by using a multi-attach volume-type. Some Block Storage API requests that used to work will be rejected with a 400 (Bad Request) response code and an error message.

Before this update, an incorrect backport resulted in an edge case where images that require additional special type handling might fail to have their headers properly processed. As a result, Red Hat Enterprise Linux CoreOS (RHCOS) 8.6 deployments failed on RHOSP 16.2.

With this update, you can now deploy RHCOS 8.6 based images using RHOSP 16.2. The backport was fixed so that the special pointer handling code can successfully fall back and understand the on-disk format supplied by RHCOS 8.6.

This update improves the way that the L3 scheduler handles OVN logical router port (LRP) assignments that cannot be correctly completed because there are no gateway chassis available in the availability zone (AZ).

Previously, if you attempted to assign an LRP to an AZ that did not have a gateway chassis, the L3 scheduler incorrectly assigned the LRP to a random set of chassis.

Now the L3 scheduler writes a warning to the logs and assigns the port to a chassis with the warning label “neutron-ovn-invalid-chassis”. There are no changes to existing LRP-to-chassis assignments.

This update introduces a Technology Preview of the security group logging feature in RHOSP 16.2.6. To monitor traffic flows and attempts into and out of an instance, you can configure Networking service packet logging for security groups.

You can associate any instance port with one or more security groups and define one or more rules for each security group. For example, you can create a rule to drop inbound SSH traffic to any instance in the finance security group. You can create another rule to allow instances in that group to send and respond to ICMP (ping) messages. Then you can configure packet logging to record combinations of accepted and dropped packet flows.

In 16.2, you can use security group logging for stateful security groups. Logged events are stored on the Compute nodes that host the instances, in the file /var/log/containers/stdouts/ovn_controller.log.

For more information about Technology Preview features, see Scope of Coverage Details.

For more information about security group logging known issues and workarounds, see https://bugzilla.redhat.com/show_bug.cgi?id=2241184 and https://bugzilla.redhat.com/show_bug.cgi?id=2192918.

Using comma-separated role-specific values for NeutronBridgeMappings results in incorrect configuration. Workaround: Specify the mappings by using an array in a YAML file instead of a comma-separated value. For example:

ComputeParameters:
  NeutronBridgeMappings:
    - datacentre:br-ex
    - datacentre2:br-ex2

A security group logging enhancement introduces a potential race condition that can generate HTTP errors when you create or delete multiple security group logs simultaneously or in rapid succession. The error happens when delete and create requests attempt to delete and create logs in the OVN database at the same time.

Workaround: To avoid this issue, send security group log create and delete requests sequentially instead of in parallel. If there are still errors, add timeouts in between requests.

In OpenStack environments with ML2/OVN, the north/south traffic for VMs with FIPs, belonging to VLAN project (tenant) networks, is centralized instead of distributed, even with DVR enabled.

This issue will be fixed in a future release of Fast Datapath (BZ 2007120). Customers experiencing this issue who need an earlier solution should request a hotfix.

In this release of Red Hat OpenStack Platform (RHOSP), there is a known issue where SR-IOV interfaces that use Intel X710 and E810 series controller virtual functions (VFs) with the iavf driver can experience network connectivity issues that involve link status flapping. The affected guest kernel versions are:

In RHOSP 16.2.5, there is a known issue where SQLAlchemy queries don’t retrieve all of the RHOSP Networking service (neutron) RBAC entries for network resources.

For example, if a network has two RBAC registers, one with action access_as_shared and other with access_as_external, and both are accessible to a specific non-admin user, the openstack port list --share query does not show the related network. However, this network is still available to the user because of the access_as_external RBAC entry.

Workaround: Currently, there is no workaround.

If your Red Hat OpenStack Platform (RHOSP) deployment uses Cisco ACI or other third-party ML2 mechanism drivers that rely on neutron-dhcp-agent, you must add DhcpAgentNotification: true to your custom heat templates before updating to RHOSP 16.2.5.

In RHOSP 16.2.5, DHCP agent notification is now disabled by default. This configuration change facilitates easier deployment of RHOSP with the ML2/OVN mechanism driver, because RHOSP ML2/OVN deployments do not use the DHCP agent. The heat templates used in ML2/OVS deployments were also modified to provide uninterrupted use of DHCP services in ML2/OVS deployments.

If your deployment uses the OVN or OVS ML2 mechanism driver, your updated deployment will not be affected.

Some third-party mechanism drivers use DHCP and rely on the DHCP notifications. If your environment uses one of these third-party mechanism drivers and you update to RHOSP 16.2.5 without enabling DHCP notification (DhcpAgentNotification: true), your updated RHOSP deployment is subject to the following issues:

The technology preview support added in RHOSP 16.1 for configuring NVDIMM Compute nodes to provide persistent memory for instances has been deprecated in RHOSP 16.2.5, and will be removed in RHOSP 17.0. Red Hat is removing support for persistent memory from RHOSP 17.0 and future releases in response to the announcement by the Intel Corporation on July 28, 2022 that they are discontinuing investment in their Intel® Optane™ business:

Cloud operators must ensure that no instances use the vPMEM feature before upgrading to 17.1.

This update fixes a bug that prevented the ceilometer-agent-compute service from collecting libvirt-related metrics.

Previously, the libvirt service started after the ceilometer-agent-compute service, which resulted in "Permission denied" failures and loss of metrics data. Now the libvirt service starts before the ceilometer-agent-compute service and the service can properly collect metrics.

Previously, Red Hat Ceph Storage nodes were incorrectly configured to consume OpenStack high availability, advanced-virt, and fast-datapath repos during Leapp upgrades. The previous bug fix for this issue introduced an override that caused role-based parameters to work incorrectly.

With this update, the role-based parameter implementation is fixed and the correct repositories are enabled for Red Hat Ceph Storage nodes. This update fixes the issue in Red Hat OpenStack Platform environments 16.2 and later that use the Red Hat Ceph Storage role.

This RHOSP 16.2.4 update makes it possible for you to correct a libvirt version incompatibility before updating to RHOSP 16.2.4. If your deployment has the incompatibility issue and you do not perform the steps published in the KCS article workaround before updating to RHOSP 16.2.4, the update might leave instances in an unmanageable state.

Before updating to 16.2.4, see the KCS article Workaround for a libvirt version-compat issue (bug 2109350) when updating RHOSP 16.2.0.

Perform the steps in the article to determine whether your update path is affected by the libvirt incompatibility issue. If it is affected, perform the remaining steps in the KCS article to resolve the issue.

This update fixes a bug that causes connectivity loss after certain updates to RHOSP 16.2.2 and 16.2.3. If you are planning to update to a RHOSP 16.2 release, update to RHOSP 16.2.4 to avoid connectivity loss.

The bug is triggered by a database schema change in OVN 21.12, which is introduced in RHOSP 16.2.2. and 16.2.3. OVN 21.12 contains a new column that is not present in earlier versions. OVN database schema changes should not cause a problem in OpenStack, but this particular change is affected by a bug.

In particular, instance connectivity is lost for a variable amount of time (from 20 seconds to 3 minutes) when you run the following command:

$ openstack overcloud external-update run --stack overcloud --tags ovn

To avoid the bug, do not update to RHOSP 16.2.2. or 16.2.3. Update to RHOSP 16.2.4 instead.

This update fixes the following PowerMax Block Storage (cinder) driver issues:

This update fixes a bug that caused ceilometer-agent-ipmi to write log data inside the container namespace instead of on the host as expected.

The improper placement of the content inside the container increased the container size, prevented proper log rotation, and resulted in loss of the log data when the container was deleted or rebuilt.

Now ceilometer-agent-ipmi writes the logs to the host in /var/log/containers/ceilometer/ as expected.

This enhancement adds new configuration options for the Networking service (neutron) logging service plug-in. With this update, there is added support for network security group logging, and you can configure Networking service logging by using the following new parameters in RHOSP Orchestration service (heat) templates:

With this update, you can use Distributed Virtual Routing (DVR) to route traffic to VLAN project networks in an ML2/OVN deployment.

Previously, if you had VLAN tenant networks and DVR in an ML2/OVN deployment, the Networking service routed traffic centrally. Now, with DVR enabled, traffic routed to the VLAN networks goes directly to a node hosting ports.

In previous releases, administrators had to add the ceph plugin to the CollectdExtraPlugins parameter in their custom environment files.

With this release, the ceph plugin loads automatically on Ceph Storage nodes. Therefore, before you upgrade from Red Hat OpenStack Platform 13 to 16.2, you must remove the ceph plugin from the CollectdExtraPlugins parameter in your custom environment files.

There is currently a known issue that the OVS minimum bandwidth value is not cleared from the port when a user removes the network policy. The workaround is to manually delete the Queue register from the OVS database:

$ ovs-vsctl destroy Queue <queue_uuid>

You can locate the Queue register by looking for the Queue external_ids:port reference, that contains the Neutron port ID.

You can cause framing errors if you configure an ID value longer than 62 characters for the metrics_qdr service. An example error message is failed: amqp:connection:framing-error connection aborted. When the metrics_qdr service is unstable, no telemetry data flows to Service Telemetry Framework (STF).

Workaround: Do not set the metrics_qdr ID value longer than 62 characters. The default value for the router ID is Router.<fqdn>, where <fqdn> is the fully-qualified domain name of the node.

Before this update, manual attempts to trigger introspection of a bare metal node that had previously failed would return the following error message:

"Can not transition from state 'uninitialized' on event 'sync'".

This was due to a defect in the Bare Metal Provisioning inspector (ironic-inspector) internal state tracking. The Bare Metal Provisioning inspector was blocking attempts to re-introspect a previously failed bare metal node because the internal state cache indicated that the session was already underway. With this update, new introspection requests explicitly reset the state cache if an entry already exists, and introspection operations that have previously failed can now be retried.

Before this update, during the replacement of a DCN node, the etcd service on the replacement node failed to start and caused the cinder-volume service on that node to fail. This failure was caused by the replacement for a DCN node attempting to start the etcd service as if it were bootstrapping a new etcd cluster, instead of joining the existing etcd cluster.

With this update, a new parameter has been added, EtcdInitialClusterState. When EtcdInitialClusterState is set to existing, the DCN node starts etcd correctly, which causes the cinder-volume service to run successfully.

Before this update, attempts to stop the cinder-volume service running in active-active mode resulted in the failed state. The cause for these failures was that the service was not allowing sufficient time to properly stop itself.

With this update, the time period allocated to stopping the service has been extended, and the service moves to the inactive state when you stop it.

Before this update, the RHOSP Bare Metal service (ironic) could lose its connection to the remote Redfish baseboard management controller (BMC) resulting in the bare metal node entering a maintenance state and with its power status changing to None. Depending on environmental factors for a site, some or all of the bare metal nodes could be in this unwanted maintenance state for an extended period of time.

Transient network connectivity issues caused by high packet loss to the BMC caused connection caching issues when using Redfish. In cases where a session token needed to be renegotiated, the cached session object was never invalidated and connectivity was lost to the BMC.

With this update, the Bare Metal service now initializes an entirely new cached session with a remote Redfish BMC when connectivity or authentication issues are detected. Additionally, this enables you to use updated credentials if the BMC passwords for the nodes are changed in the future.

In Red Hat OpenStack Platform (RHOSP) 16.2.3, Intel Columbiaville E810 NICs in NFV deployments are now fully supported with the following known issues:

In Red Hat OpenStack Platform (RHOSP) 16.2.3, there is support for upgrading firmware and configuring NVIDIA Mellanox BlueField-2 into ConnectX mode by using the mstflint tool, with these two known issues:

In Red Hat OpenStack Platform (RHOSP) 16.2.3, a technology preview is available that supports Open vSwitch (OVS) Traffic Control (TC) Flower classifier hardware offload using connection tracking (conntrack), with the following known issue:

With RHOSP 16.2.3, fencing agents that depend on potentially insecure protocols, such as Telnet, have been removed from the default overcloud images. Overcloud images now include the following fencing agents by default:

Open vSwitch (OVS) does not support offloading OpenFlow rules that have the skb_priority, skb_mark, or output queue fields set. Those fields are needed to provide quality-of-service (QoS) support for virtio ports.

If you set a minimum bandwidth rule for a virtio port, the Neutron Open vSwitch agent marks the traffic of this port with a Packet Mark Field. As a result, this traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means no traffic can be offloaded.

As a workaround, if your environment includes OVS hardware offload ports, disable the packet marking in the nodes that require hardware offloading. After you disable the packet marking, it will not be possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules will still be available.

In the configuration file, set the disable_packet_marking flag to true. After you edit the configuration file, you must restart the neutron_ovs_agent container. For example:

$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
  [ovs]
  disable_packet_marking=True

Rebooting a node with a virtual function (VF) attached to OVS-DPDK (vfio-pci driver) results in VF uninitialized on that physical function (PF). As a result, virtual machines are unable to use the VFs from that PF. If a second VF is used for another OSP network, it does not function as expected after reboot.

As a workaround, perform the following steps on the Compute node before you reboot the node:

There is currently a known issue in which the OpenStack tripleo validator run command produces errors. In the command output the Status_by_Host column contains the value, 'No host matched'. This error occurs with all openstack validator run calls because no ansible inventory was available for the undercloud and is not limited to a single validator group. This error also impacts overcloud validations. The root cause of this is a regression in the tripleo client code where tripleo-ansible-inventory is no longer called in real time.

As a result, in updates to Red Hat OpenStack Platform (RHOSP) 16.2, the tripleo validator run --group pre-introspection command fails on all tests.

Workaround: create a file called inventory.yaml by manually running tripleo-ansible-inventory --static-yaml-inventory inventory.yaml. Then, run the validation using the -i inventory.yaml argument:

# tripleo-ansible-inventory --static-yaml-inventory inventory.yaml
# openstack tripleo validator run  --group pre-introspection -i inventory.yaml

For more information, see the Red Hat Knowledgebase solution openstack tripleo validator run command produces error.

There is currently a known issue that the OVS minimum bandwidth value is not cleared from the port when a user removes the network policy. The workaround is to manually delete the Queue register from the OVS database:

$ ovs-vsctl destroy Queue <queue_uuid>

You can locate the Queue register by looking for the Queue external_ids:port reference, that contains the Neutron port ID.

Before this update, the leapp process failed when the following statements were true:

Before this update, when creating a snapshot with PowerMaxOS 5978.711, REST experienced a payload response change and caused the device label to modify its format. The underlying data from the solutions enabler changed and no longer contained a colon character (:). This caused an IndexError exception in the PowerMax Driver:

IndexError: list index out of range

With this update, the problem is resolved in PowerMaxOS 5978.711 and later.

In cases where high CPU use was monitored in a multi-core system, the calculation for CPU use was inaccurate.

With this update, the calculation of CPU use in a multi-core scenario is now accurate. The latest STF dashboards have been adjusted to incorporate this update.

This update fixes a bug that caused unintended results when converting Dell EMC PERC H755 RAID controller physical disks to non-RAID mode.

The conversion erroneously created RAID-0 virtual disks and moved them to the Online state, consuming a physical disk.

The RAID-0 virtual disks are no longer created during the conversions.

This update fixes a bug that caused some virtual machine (VM) bootup operations to fail when multiple VMs were booted simultaneously from an image.

Previously, the Block Storage service (cinder) GPFS SpectrumScale driver did not correctly detect when the storage back end supported copy-on-write (COW) mode. The driver disabled COW features such as the ability to rapidly create volumes from an image. When booting multiple instances simultaneously from an image, some instances timed out when copying the image to its boot volume.

The GPFS Block Storage service driver now properly detects when the storage back end supports COW mode. You can now spawn multiple VM instances simultaneously.

This update fixes a bug that prevented connections to the Ceph storage backend with Ceph client release 15.2.0 (Octopus) and later, affecting Red Hat Ceph Storage 5.0 and later.

A temporary configuration file generated to enable a Ceph connection did not include a '[global]' section marker. This update adds the '[global]' section marker to the temporary file.

The section marker was introduced in Ceph client release 0.94.0 (Hammer). Starting with the Octopus release, Ceph requires the presence of the marker. This fix is backward compatible to Red Hat Ceph Storage 4.x.

Red Hat OpenStack Platform (RHOSP) does not support the use of a fully qualified domain name (FQDN) as the instance display name in a boot server request. The instance display name is passed from the boot server request to the instance.hostname field.

A recent update now sanitizes the instance.hostname field. The sanitization steps include replacing periods with dashes, a replacement that makes it impossible to continue using the unsupported FQDN instance display names.

This update provides a temporary workaround for customers who use a fully qualified domain name (FQDN) as the instance display name in a boot server request. It limits the scope of the sanitization to cases where the instance display name ends with a period followed by one or more numeric digits.

If you use FQDN as the instance display name in a boot server request, modify your workflow before upgrading to RHOSP 17.

The Block Storage service (cinder) can now use an external NFS share to perform image format conversion of Image service (glance) images on the overcloud Controller nodes. Using this functionality prevents the space on the node from being completely filled during a conversion operation.

See Configuring an external NFS share for conversion.

This enhancement enables the experimental rsyslog reopenOnTruncate setting to ensure that rsyslog immediately recognizes when a logrotation happens on a file. The setting affects every service configured to work with rsyslog.

When rsyslog reopenOnTruncate is disabled, rsyslog waits for a log file to fill to its original capacity before consuming any additional logs.

With this update, you can set QoS maximum bandwidth limit, egress direction rules on hardware-offloaded ports in a ML2/OVS deployment. To set the policy, use the normal QoS policy/rules methods.

The back end uses ip link commands to enforce the policy instead of the normal OVS QoS engine, because the OVS meter action cannot be offloaded. See meter action is not offloaded.

With this technology preview update, you can set the following parameters to configure OVS PMD automatic load balance:

Starting with Red Hat Ceph Storage 4, you can enable encryption for all traffic generated by the Ceph daemons over the network.

The secure mode setting for messenger v2 encrypts the communication between Ceph daemons and Ceph clients, effecting an end-to-end encryption.

A new tripleo-heat-templates parameter can enable the on-wire encryption between daemons and clients. To configure Ceph to enable the on-wire encryption between daemons and clients, add the following lines to the overcloud deployment environment files:

parameter_defaults:
  CephMsgrSecureMode: true

Disable the advanced-virt-for-rhel-8 repository before you install Red Hat OpenStack Platform (RHOSP) 16.2, update from RHOSP 16.2 to a newer maintenance release, or upgrade from 16.1 to 16.2.

RHOSP hosts do not require the advanced-virt-for-rhel-8 repository. If you do not disable it, dependency issues cause the installation, update, or upgrade to fail. The dependency failures happen because the advanced-virt-for-rhel-8-x86_64-rpms repository is based on RHEL 8.latest, which does not work with RHEL 8.4.

As a workaround, disable the repositories. Perform the steps appropriate for your installation, update, or upgrade scenario.

There is currently a known issue in which the OpenStack tripleo validator run command produces errors. In the command output the Status_by_Host column contains the value, 'No host matched'. This error occurs with all openstack validator run calls because no ansible inventory was available for the undercloud and is not limited to a single validator group. This error also impacts overcloud validations. The root cause of this is a regression in the tripleo client code where tripleo-ansible-inventory is no longer called in real time.

As a result, in updates to Red Hat OpenStack Platform (RHOSP) 16.2, the tripleo validator run --group pre-introspection command fails on all tests.

Workaround: create a file called inventory.yaml by manually running tripleo-ansible-inventory --static-yaml-inventory inventory.yaml. Then, run the validation using the -i inventory.yaml argument:

# tripleo-ansible-inventory --static-yaml-inventory inventory.yaml
# openstack tripleo validator run  --group pre-introspection -i inventory.yaml

For more information, see the Red Hat Knowledgebase solution openstack tripleo validator run command produces error.

Before this update, the certmonger package was dropped from the minimal image, which made it impossible to deploy Red Hat Ceph Storage nodes.

With this update, the certmonger package has been added back to the image, and Red Hat Ceph Storage nodes can now be deployed.

In Red Hat OpenStack Platform (RHOSP) 16.2.1, a technology preview is available that supports Intel Columbiaville E810 NICs in NFV deployments, with the following recommendations:

RHOSP does not yet support ML2/OVN with VXLAN networks. The migration process includes steps to convert VXLAN networks to Geneve. When the migration target version is RHOSP 16.2.0, a bug prevents the expected VXLAN to Geneve conversion, and the networks remain configured as VXLAN.

This bug affects only migrations to ML2/OVN on RHOSP 16.2. It does not affect migrations to ML2/OVN on RHOSP 16.1.

Typically, when you create an encrypted volume from a snapshot of an encrypted volume, the source volume is the same size or smaller than the destination volume.

In previous releases, if you created an encrypted volume from a snapshot of an encrypted volume, and the destination volume was close to or equal to the size of the source volume, the Block Storage service (cinder) silently truncated the data in the new destination volume.

With this release, the Block Storage service calculates the size of the destination volume to include the current size of the encryption header, which eliminates the data truncation.

Previously, the Shared File Systems service (manila) API that brings external shares into service management did not check for duplicated export locations. An existing share brought into the service multiple times results in an inconsistent state.

With this release, the API evaluates the export locations of known or existing shares before allowing external shares to be managed, and prevents existing shares from being erroneously brought into the Shared File Systems service again.

When an instance is created, the Compute service (nova) sanitizes the instance display name to generate a valid hostname when DNS integration is enabled in the Networking service (neutron).

Before this update, the sanitization did not replace periods ('.') in instance names, for example, 'rhel-8.4'. This could result in display names being recognized as Fully Qualified Domain Names (FQDNs) which produced invalid hostnames. When instance names contained periods and DNS integration was enabled in the Networking service, the Networking service rejected the invalid hostname, which resulted in a failure to create the instance and a HTTP 500 server error from the Compute service.

With this update, periods are now replaced by hyphens in instance names to prevent hostnames being parsed as FQDNs. You can continue to use free-form strings for instance display names.

Before this update, the NetApp SolidFire driver created a duplicate volume when the API response is lost due to a connection error and the driver retries the API request. This occurred when the SolidFire back end successfully received and processed a create volume operation, but failed to deliver the response back to the driver. This update resolves the issue by:

Before this update, the Shared File Systems service (manila) dashboard had dynamic form elements whose names could potentially cause the forms to become unresponsive. This meant that the creation of share groups, share networks, and shares within share networks did not work.

With this update, dynamic elements whose names could be problematic are encoded, which means that creation of share groups, share networks, and shares within share networks functions normally.

The Shared File Systems service (manila) uses the CephFS volume client to communicate with Ceph Storage clusters. Previously, the CephFS volume client package aborted while creating or deleting file systems.

The aborted operations caused the manila-share process within the Shared File Systems service to restart, which caused shares that were being provisioned or deleted to be stuck in creating or deleting states, respectively.

With this release, the CephFS volume client package no longer aborts provisioning or deletion requests, and the manila-share process does not restart during these operations.

This update fixes an issue that caused Networking service (neutron) agents, such as Networking service DHCP, to fail when they tried to create resources in OVN. This was caused by residual data left in the OVN databases when QoS rules were created for floating IPs.

This update eliminates the residual data and fixes the problem.

This update fixes a known issue where the Open Virtual Network (OVN) Metadata service was not available to VM instances bound to an SR-IOV virtual function. The issue did not affect network function but these instances did not receive their SSH keys in the absence of a Metadata service connection.

The metadata service connectivity for SR-IOV ports now functions correctly.

Before this fix, grub2 tooling wrote kernel argument changes to /boot/grub2/grubenv. This file was not available to UEFI boot systems, and caused kernel argument changes not to persist across reboots on UEFI boot nodes.

This fix changes both the /boot/grub2/grubenv file and the /boot/efi/EFI/redhat/grubenv files when you make kernel argument changes. As a result, RHOSP director now applies persistent Kernel argument changes for UEFI boot nodes.

During stack update the KernelArgs could be modified or appended. You must perform a reboot of the affected nodes manually.

For example, if the current deployment has the following configuration, it is possible to change hugepages=64, or add or remove arguments during the stack update:

`KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 intel_iommu=on iommu=pt isolcpus=1-11,13-23"

For example:

KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt isolcpus=1-24"
KernelArgs: "isolcpus=1-11,13-23"

With this enhancement, you can improve the performance of live migrations by using the following new parameters:

The Red Hat OpenStack Platform (RHOSP) director Operator creates a set of custom resource definitions (CRDs) on top of Red Hat OpenShift Container Platform to manage resources normally created by the RHOSP undercloud. CRDs are split into two types for hardware provisioning and software configuration. The operator includes CRDs to create and manage overcloud networks, manage IP addresses, create VM sets for RHOSP Controllers, and create bare metal sets for RHOSP Computes.

For Technology Preview, the software configuration is accomplished with an OpenStackClient pod, which uses traditional RHOSP or TripleO interfaces and CLI commands. Work is ongoing to produce a more scalable Heat-to-Ansible playbook deployment workflow within the RHOSP director Operator.

For Image Service (glance) image conversion, the glance-direct method is not enabled by default. To enable this feature, set enabled_import_methods to [glance-direct,web-download] or [glance-direct] in the DEFAULT section of the glance-api.conf file.

The Image Service must have a staging area when you use the glance-direct import method. Set the node_staging_uri option in the DEFAULT section of the glance-api.conf file to file://<absolute-directory-path>. This path must be on a shared file system that is available to all Image service API nodes.

Starting with Red Hat Ceph Storage 4, you can enable encryption for all traffic generated by the Ceph daemons over the network.

The secure mode setting for messenger v2 encrypts the communication between Ceph daemons and Ceph clients, effecting an end-to-end encryption.

A new tripleo-heat-templates parameter can enable the on-wire encryption between daemons and clients. To configure Ceph to enable the on-wire encryption between daemons and clients, add the following lines to the overcloud deployment environment files:

parameter_defaults:
  CephMsgrSecureMode: true

In previous releases, administrators had to add the ceph plugin to the CollectdExtraPlugins parameter in their custom environment files.

With this release, the ceph plugin loads automatically on Ceph Storage nodes. Therefore, before you upgrade from Red Hat OpenStack Platform 13 to 16.2, you must remove the ceph plugin from the CollectdExtraPlugins parameter in your custom environment files.

Significant connectivity loss can affect your workloads after certain updates to RHOSP 16.2.2 and 16.2.3.

The bug affects your deployment when you update RHOSP to 16.2.2 or 16.2.3 from 16.2.0, 16.2.1, or from any 16.1 release. To verify if your update path is affected, see the article [1].

If your planned update path is affected by the bug and you must update now to 16.2.2 or 16.2.3, contact your Red Hat support representative to see if your deployment is compatible with a Hot Fix that addresses the bug.

In particular, instance connectivity is lost for a variable amount of time (from 20 seconds to 3 minutes) when you run the following command:

$ openstack overcloud external-update run --stack overcloud --tags ovn

A fix for this bug is scheduled for RHOSP 16.2.4.

[1] Follow the steps in this article to determine if your update path is affected:

https://access.redhat.com/solutions/6974338

Do not update from RHOSP 116.2.0 to 16.2.2 or 16.2.3 until you evaluate your risk of serious impact from a libvirt version incompatibility. To complete this evaluation, check the libvirt package in the nova_libvirt container in all compute nodes:

$ sudo podman exec nova_libvirt rpm -q libvirt

If the libvirt version is 7.0, the deployment IS NOT affected by the bug. You can perform the update.

If the libvirt version is 7.6, the deployment is affected by the buy. Your update is at risk.

If you learn that your update is at risk from the libvirt incompatibility, choose one of these options:

Wait: Update directly to RHOSP 16.2.4 when it is released. It includes a fix for the incompatibility issue. This is the preferred option if you can postpone the update. In 16.2.4 you can perform the manual steps outlined in KCS [1] without any hot fix, because the option skip_hypervisor_version_check_on_lm is included in 16.2.4.

Hot fix: Contact your Red Hat support representative to explore whether your environment is compatible for a hot fix patch that resolves the issue. Use this option if there is a strong business need for an immediate update.

If you already performed the update with the version incompatibility, see the KCS article [2] for guidance on fixing the problem.

[1] Article helps you verify if your update is at risk (same step as above). Also includes steps you can perform to update to 16.2.1, 16.2.2, or 16.2.3 if you choose the Hot Fix option. See https://access.redhat.com/solutions/6972451

[2] Article helps you fix your deployment if you already completed an update to 16.2.1, 16.2.2, or 16.2.3 and your deployment is affected by the incompatibility. See https://access.redhat.com/solutions/6969430

Starting with Red Hat Enterprise Linux (RHEL) version 8.3, support for the Intel Transactional Synchronization Extensions (TSX) feature is disabled by default. Currently, this causes instance live migration to fail when migrating from hosts where the TSX kernel argument is enabled to hosts where the TSX kernel argument is disabled.

This impact applies only to Intel hosts that support the TSX feature. For more information about the CPUs that are affected by this issue, see Affected Configurations.

For more information, see the Red Hat Knowledgebase solution Guidance on Intel TSX impact on OpenStack guests.

In Red Hat OpenStack Platform (RHOSP) deployments that use the Modular Layer 2 plug-in with the Open vSwitch (ML2/OVS) mechanism driver, there is currently a known issue where the Orchestration service (heat) parameter, NeutronL3AgentAvailabilityZone does not set the relevant Neutron L3 agent parameter correctly.

Workaround: use a custom hieradata statement to set this value. In the example that follows, replace [ROLE] with the composable role name that is appropriate for your site.

[ROLE]ExtraConfig:
    neutron::agents::l3::availability_zone: role_availability_zone

For more information, see Puppet: Customizing hieradata for roles in the Advanced Overcloud Customization guide.

Rebooting a node with a virtual function (VF) attached to OVS-DPDK (vfio-pci driver) results in VF uninitialized on that physical function (PF). As a result, virtual machines are unable to use the VFs from that PF. If a second VF is used for another OSP network, it will not function as expected after reboot.

Workaround: perform the following steps on the Compute node before you reboot the node:

RHOSP does not yet support ML2/OVN with VXLAN networks. The migration process includes steps to convert VXLAN networks to Geneve. When the migration target version is RHOSP 16.2.0, a bug prevents the expected VXLAN to Geneve conversion, and the networks remain configured as VXLAN.

This bug affects only migrations to ML2/OVN on RHOSP 16.2. It does not affect migrations to ML2/OVN on RHOSP 16.1.

There are known limitations for Mellanox ConnectX-5 adapter cards in VF link aggregation group (LAG) mode in OVS OFFLOAD deployments, SRIOV Switchdev mode.

When at least one VF of any physical function (PF) is still bound or attached to a VM, an internal firmware error occurs when attempting to disable single-root input/output virtualization (SR-IOV), and when unbinding PF using a function such as ifdown and ip link.

Workaround: Reboot the node to restore the bond. For more information, see the Red Hat Knowledgebase solution Mellanox ConnectX-5 internal error when removing PF from the bond or disabling SR-IOV.

VF LAG mode with OVS OFFLOAD, SRIOV switchdev mode is not supported, if you use the mstconfig tool to set a value higher than 64 for the NUM_OF_VFS parameter in the Firmware configuration. Currently, there is no workaround available.

In Red Hat Openstack Platform (RHOSP) 16.2, support for the QXL video model is deprecated, due to the removal of support for the Spice graphics software in RHEL 9. This will cause issues for instances that use QXL when migrating from RHEL-8 to RHEL-9. Red Hat recommends using the virtio video model for both UEFI and BIOS instances, instead of qxl. When creating a new instance from an image, set the video model before launching the new instance:

$ openstack image set --property hw_video_model=virtio <image>

To update the video model for existing instances that use the QXL video model: