Chapter 3. Release information

With this update, Red Hat support for Multi-RHEL is expanded to include the following deployment architectures:

Before this update, libvirt-related containers on Compute nodes were needlessly restarted during deployment, update, and scaling operations, even when the container configurations were not changed.

With this update, libvirt-related containers that do not have configuration changes are not restarted during deployment, update, and scaling operations.

Before this update, after every successful Red Hat Ceph Storage adoption during upgrade, the ceph-ansible package was removed by default.

This update introduces the tag, cleanup_cephansible, to the task that removes ceph-ansible. You can use this tag with --skip-tags while running the adoption playbook to avoid removal.

Before this update, during a DCN FFU system upgrade of nodes on the setup with multiple stacks, the Red Hat Ceph Storage task Set noout flag might fail to run the ceph command on the right host.

After the update, a system upgrade on any node in a multi-stack setup now delegates the Red Hat Ceph Storage task Set noout flag to the relevant host, and the ceph commands are run on the specific cluster.

This update fixes a bug that caused unintended deletion of Load-balancing service (octavia) health monitor ports and OVN metadata ports.

Before this update, in RHOSP environments that used Load-balancing service health monitor ports from a previous version, running neutron-db-sync-tool sometimes randomly deleted those pre-existing ports or OVN metadata ports. This unintended port deletion resulted in loss of health monitor capacity or communication loss with the affected instances.

This update resolves the conflicts. It uses a new value, ovn-lb-hm:distributed, for the OVN Load-balancing service health monitor ports device_owner field. Old OVN Load-balancing service health monitor ports are automatically updated with this version.

Before this update, a director bug could disrupt or crash client workloads during updates or upgrades to any RHOSP 17.1 version. This bug affected deployments that enabled the RHOSP Shared File System service (manila) with the CephFS on NFS back ends.

With this update this issue has been addressed, and the Shared File System service (manila) operates properly when users set up "access rules" on their shares.

Before this update, the FFU procedure sometimes used the incorrect Red Hat Ceph Storage image when the documented upgrade procedure was not followed by the user.

With this update, the FFU procedure always uses the correct Red Hat Ceph Storage image. The multi-rhel-container-image-prepare.py script has been updated to use the correct defaults and version validation checks have been added to the FFU process.

This update fixes a bug that prevented load-balancing of traffic on some Load-balancing service (octavia) pool members on IPv6 networks in an ML2/OVN environment.

Before this update, if you added a second listener+pool+member to a pool, the pool entered an ERROR state and traffic within that pool was not load-balanced.

With this update, traffic is load-balanced to all members as expected.

This update safeguards against upgrades from RHOSP 16.2 to RHOSP 17.1 with libvirt configurations that could result in workload disruptions after the upgrade.

Before this update, if you performed an upgrade to RHOSP 17.1 from a RHOSP 16.2 environment that included a modular deployment of libvirt on Red Hat Enterprise Linux (RHEL) 8 or that ran libvirt UBI9 on RHEL 8, these configurations sometimes resulted in workload disruption.

With this update, the upgrade from RHOSP 16.2 to 17.1 fails if the RHOSP 16.2 environment includes a modular deployment of libvirt on Red Hat Enterprise Linux (RHEL) 8, or runs libvirt UBI9 on RHEL 8.

This update fixes a bug that prevented operation of the OVN Health Monitoring for Load-balancing service (octavia) on IPv6 networks in ML2/OVN deployments.

Previously, the OVN Health Monitors service did not correctly identify the ONLINE and OFFLINE statuses of back-end members.

With this update, load-balancing works as expected, and the OVN Health Monitors correctly identify the ONLINE and OFFLINE statuses of back-end members.

Before this update, during upgrades to RHOSP 17.1 and updates between minor versions of 17.1, Networking service (neutron) environments that used the ML2/OVN mechanism driver stopped receiving updates from the OVN databases for a period of time. When the Controller node that contained the RAFT leader was also updated, the mechanism driver would then receive the updates from the OVN databases.

With this update, this problem has been fixed. Now, during RHOSP updates and upgrades, the Networking service using the ML2/OVN mechanism driver operates properly.

RHOSP 17.1.3 now supports a new aging mechanism for MAC addresses learned through the localnet_learn_fdb option that is included in Open Virtual Network (OVN) version 23.09. This new aging mechanism consists of two new options, fdb_age_thhreshold and fdb_removal_limit.

The fdb_age_thhreshold option enables you to set the maximum time the learned MACs stay in the FDB table (in seconds).

The fdb_removal_limit option prevents OVN from removing a large number of FDB table entries all at one time.

When you use these new options with localnet_learn_fdb, you reduce the likelihood of performance and scalability issues caused by the FDB table from growing too big, a problem often experienced in RHOSP environments that have large provider networks.

In RHOSP 17.1, a technology preview is available for the VF-LAG transmit hash policy offload that enables load balancing at NIC hardware for offloaded traffic/flows. This hash policy is only available for layer3+4 base hashing.

To use the technology preview, verify that your templates include a bonding options parameter to enable the xmit hash policy as shown in the following example:

bonding_options: "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer3+4"

Currently, in RHOSP 17.1 environments that use BGP dynamic routing, the RHOSP Compute service cannot route packets sent from one of these instances to a multicast IP address destination. Therefore, instances subscribed to a multicast group fail to receive the packets sent to them. The cause is that BGP multicast routing is not properly configured on the overcloud nodes.

Workaround: Currently, there is no workaround.

Adding a load balancer member whose subnet is not in the Load-balancing service (octavia) availability zone puts the load balancer in ERROR. The member cannot be removed because of the ERROR status, making the load balancer unusable.

Workaround: Delete the load balancer.

In RHOSP environments with ML2/OVN or ML2/OVS that have DVR enabled and use VLAN tenant networks, east/west traffic between instances connected to different tenant networks is flooded to the fabric.

As a result, packets between those instances reach not only the Compute nodes where those instances run, but also any other overcloud node.

This might impact the network and it might be a security risk because the fabric sends traffic everywhere.

This bug will be fixed in a later FDP release. You do not need to perform a RHOSP update to obtain the FDP fix.

Currently, the Retbleed vulnerability mitigation in RHEL 9.2 can cause a performance drop for Open vSwitch with Data Plane Development Kit (OVS-DPDK) on Intel Skylake CPUs.

This performance regression happens only if C-states are disabled in the BIOS, Hyper-Threading Technology is enabled, and OVS-DPDK is using only one logical core of a given core.

Workaround: Assign both logical cores to OVS-DPDK or to SR-IOV guests that have DPDK running as recommended in Configuring network functions virtualization.

RHOSP 17.1 with the OVN mechanism driver does not support logging of flow events per port or the use of the --target option of the network log create command.

RHOSP 17.1 supports logging of flow events per security groups, using the --resource option of the network log create command. For more information, see Logging security group actions in Configuring Red Hat OpenStack Platform networking.

The Networking service (neutron) does not prevent you from disabling or removing a networking profile, even if that profile is part of a flavor that is in use by a router. The disablement or removal of the profile can disrupt proper operation of the router.

Workaround: Before you disable or remove a networking profile, ensure that it is not part of a flavor that is currently used by a router.

The Ceph Dashboard cannot reach the Prometheus service endpoint and displays the following error message: 404 not found. This error occurs because the configuration of the VIP for the Prometheus service is not correct.

Workaround:

If your deployment has an external Ceph cluster with multiple file systems, you can not create a Shared File System service (Manila) share as expected.

The cephfs_filesystem_name driver configuration parameter that is needed to avoid this situation cannot be set using director’s heat template parameters.

Workaround: Set the "cephfs_filesystem_name" parameter to specify the filesystem that the Shared File System service (Manila) must use via "ExtraConfig".

Add the parameter to an environment file as shown in the following example:

$ cat /home/stack/manila_cephfs_customization.yaml
parameter_defaults:
  ExtraConfig:
    manila::config::manila_config:
      cephfs/cephfs_filesystem_name:
        value: <filesystem>

Replace the value of <filesystem> with the appropriate name and include this environment file with the openstack overcloud deploy command.

The way cgroups are managed for libvirt changed between RHEL releases. As a result, virsh cpu-stats is not an officially supported part of the Red Hat OpenStack Platform (RHOSP) product. The functionality is provided by the underlying version of libvirt received from RHEL. Greenfield RHOSP 17.x is only supported on RHEL 9, which uses cgroups v2. The cgroups v2 API does not provide the API required to support the virsh cpu-stats API and this functionality is unavailable when using 17.1 on RHEL 9.

RHOSP 17.1 on RHEL 8, supported during mixed RHEL upgrades from 16.2, has been updated to make virsh cpu-stats functional while running on rhel 8.4. As a result, virsh cpu-stats functionality has been restored on RHEL 8 hosts but is unavailable on fully upgraded RHEL 9 hosts.

When upgrading RHOSP 16.2 to 17.1 on environments that use Lenovo SR650 servers, the servers fail on their first boot, displaying a blue screen stating that there is a lack of valid boot devices.

This issue is caused by Lenovo UEFI firmware resetting boot records after deployment. RHOSP director requests two changes to UEFI firmware settings. However, Lenovo hardware can only handle one request before rebooting.

Workaround: you must manually reboot the Lenovo servers to the desired operating system.

In RHOSP 17.1 environments that use dynamic routing and the OpenStack Load-balancing service (octavia) with the OVN provider driver, there is a known issue where the Load-balancing VIP is deleted. The deletion is caused by the process that synchronizes the OVN BGP agent and the Load-balancing service.

Workaround: The workaround is to increase the reconcile interval to a very high value. Create a custom environment YAML file and add the following values:

parameter_defaults:
  FrrOvnBgpAgentReconcileInterval: 999999

For more information, see 4.11. Deploying a spine-leaf enabled overcloud.

In RHOSP environments that use dynamic routing, during a minor update, Free Range Routing (FRR) is restarted twice in a row. This occurs during updates in the following scenarios:

This update fixes an issue that disrupted connection to instances over IPv6 during migration from the OVS mechanism driver to the OVN mechanism driver.

Now you can migrate from OVS to OVN with IPv6 without experiencing the instance connection disruption.

This update fixes a bug that caused the ceph-nfs service to fail after a reboot of all Controller nodes.

The Pacemaker-controlled ceph-nfs resource requires a runtime directory to store some process data.

Before this update, the directory was created when you installed or upgraded RHOSP. However, a reboot of the Controller nodes removed the directory, and the ceph-nfs service did not recover when the Controller nodes were rebooted. If all Controller nodes were rebooted, the ceph-nfs service failed permanently.

With this update, the directory is created before spawning the ceph-nfs service, and the cephfs-nfs service continues through reboots.

This update fixes a permission issue that caused collectd sensubility to stop working after you rebooted a baremetal node.

Now collectd sensubility continues working after you reboot a baremetal node.

This update fixes an issue that sometimes caused the security group logging queue to stop accepting entries before reaching the limit set in NeutronOVNLoggingRateLimit.

You can set the maximum number of log entries per second with the parameter NeutronOVNLoggingRateLimit. If the log entry creation exceeds that rate, the excess is buffered in a queue up to the number of log entries that you specify in NeutronOVNLoggingBurstLimit.

Before this update, during short bursts, the queue sometimes stopped accepting entries before reaching the limit specified in NeutronOVNLoggingBurstLimit.

With this update, the NeutronOVNLoggingBurstLimit value affects the queue limit as expected.

Before this update, puppet-ceilometer did not populate the tenant_name_discovery parameter in the ceilometer configuration on Compute nodes. This prevented identification of the Project name and User name fields.

With this update, the addition of the tenant_name_discovery parameter to the Compute namespace in puppet-ceilometer resolves the issue. When the tenant_name_discovery parameter is set to true, the Project name and User name fields are populated.

Previously, the nova_virtlogd container did not get updated to from ubi 8 to ubi 9 as expected after a RHOSP upgrade of the Compute node to RHOSP 17.1 with RHEL 9.2. The container was updated only after rebooting the Compute node.

Now, the nova_virtlogd container gets updated to ubi 9 before the RHOSP upgrade. Note that in subsequent RHOSP updates, you must reboot the Compute node after any change to the virtlogd container, because a restart would cause workload logs to become unreachable.

Before this update, openstack overcloud deploy did not pass the value of the OVNAvailabilityZone role parameter to OVS.

With this update, the OVNAvailabilityZone role parameter correctly passes the value as an availability-zones value in external-ids:ovn-cms-options.

The following example shows how to use the parameter in an environment file to set `OVNAvailabilityZone. Include the environment file in the deployment command.

ControllerParameters:
  OVNAvailabilityZone: 'az1'

The deployment adds availability-zones=az1 to OVS external-ids:ovn-cms-options.

This update fixes a bug that set the wrong MTU value on tenant networks that were changed from VXLAN to Geneve after a migration from the OVS mechanism driver to the OVN mechanism driver. Before this update, the cloud-init package overrode the value that was correctly set by the DHCP server.

For example, after a migration from the OVS mechanism driver with VXLAN to the OVN mechanism driver to Geneve with a 1442 MTU, cloud-init reset the MTU to 1500.

With this update, the value set by the DHCP server persists.

In RHOSP 17.1, a technology preview is available for the VF-LAG transmit hash policy offload that enables load balancing at NIC hardware for offloaded traffic/flows. This hash policy is only available for layer3+4 base hashing.

To use the technology preview, verify that your templates include a bonding options parameter to enable the xmit hash policy as shown in the following example:

bonding_options: "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer3+4"

A RHOSP deployment can fail when a very large number of virtual functions (VFs) are created per physical function (PF). The NetworkManager issues a DHCP request on all of them, leading to failures in the NetworkManager service.

For example, this issue occurred during a deployment that included 256 VFs across 4 PFs.

Workaround: Avoid creating a very large number of VFs per PF.

In RHOSP 17.1 environments that use BGP dynamic routing, there is currently a known issue where floating IP (FIP) port forwarding fails.

When FIP port forwarding is configured, packets sent to a specific destination port with a destination IP that equals the FIP are redirected to an internal IP from a RHOSP Networking service (neutron) port. This occurs regardless of the protocol that is used: TCP, UDP, and so on.

When BGP dynamic routing is configured, the routes to the FIPs used to perform FIP port forwarding are not exposed, and these packets cannot reach their final destinations.

Workaround: Currently, there is no workaround.

Adding a load balancer member whose subnet is not in the Load-balancing service (octavia) availability zone puts the load balancer in ERROR. The member cannot be removed because of the ERROR status, making the load balancer unusable.

Workaround: Delete the load balancer.

In RHOSP environments with ML2/OVN or ML2/OVS that have DVR enabled and use VLAN tenant networks, east/west traffic between instances connected to different tenant networks is flooded to the fabric.

As a result, packets between those instances reach not only the Compute nodes where those instances run, but also any other overcloud node.

This might impact the network and it might be a security risk because the fabric sends traffic everywhere.

This bug will be fixed in a later FDP release. You do not need to perform a RHOSP update to obtain the FDP fix.

Currently, the Retbleed vulnerability mitigation in RHEL 9.2 can cause a performance drop for Open vSwitch with Data Plane Development Kit (OVS-DPDK) on Intel Skylake CPUs.

This performance regression happens only if C-states are disabled in the BIOS, Hyper-Threading Technology is enabled, and OVS-DPDK is using only one logical core of a given core.

Workaround: Assign both logical cores to OVS-DPDK or to SRIOV guests that have DPDK running as recommended in the NFV configuration guide.

RHOSP 17.1 with the OVN mechanism driver does not support logging of flow events per port or the use of the --target option of the network log create command.

RHOSP 17.1 supports logging of flow events per security groups, using the --resource option of the network log create command. For more information, see Logging security group actions in Configuring Red Hat OpenStack Platform networking.

Currently, there is no support for Multi-RHEL for the following deployment architectures:

In RHOSP 17.1 GA environments that use the ML2/OVN mechanism driver, floating IP port forwarding does not function correctly.

FIP port forwarding should be centralized on the Controller or the Networker nodes. Instead, VLAN and flat networks distribute north-south network traffic when FIPs are used.

Workaround: To resolve this problem and force FIP port forwarding through the centralized gateway node, either set the RHOSP Orchestration service (heat) parameter NeutronEnableDVR to false, or use Geneve instead of VLAN or flat project networks.

In this release of RHOSP, SR-IOV interfaces that use Intel X710 and E810 series controller virtual functions (VFs) with the iavf driver can experience network connectivity issues that involve link status flapping. The affected guest kernel versions are:

The metadata service can become unavailable after the metadata agent fails in multiple attempts to start a malfunctioning HAProxy child container. The metadata agent logs an error message similar to: `ProcessExecutionError: Exit code: 125; Stdin: ; Stdout: Starting a new child container neutron-haproxy-ovnmeta-<uuid>”.

Workaround: Run podman kill <_container name_> to stop the problematic haproxy child container.

The Networking service (neutron) does not prevent you from disabling or removing a networking profile, even if that profile is part of a flavor that is in use by a router. The disablement or removal of the profile can disrupt proper operation of the router.

Workaround: Before you disable or remove a networking profile, ensure that it is not part of a flavor that is currently used by a router.

The Ceph Dashboard cannot reach the Prometheus service endpoint and displays the following error message: 404 not found. This error occurs because the configuration of the VIP for the Prometheus service is not correct.

Workaround:

Some AMD environments fail to boot when provisioned with the overcloud-hardened-uefi-full.raw image, due to the included kernel argument console=ttyS0. As a result, the boot sequence halts with no diagnostic nor error message.

Workaround: Run the following commands to edit the overcloud image:

sudo yum install guestfs-tools -y

sudo systemctl start libvirtd

sudo virt-customize -a /var/lib/ironic/images/overcloud-hardened-uefi-full.raw \
    --run-command "sed -i 's/console=ttyS0 //g' /etc/default/grub" \
    --run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
    --run-command "grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg"

After running those commands, you can provision the AMD nodes using the provision command.

In RHOSP 17.1.2 environments that contain Load-balancing service (octavia) health monitor ports from a previous version, running neutron-db-sync-tool might randomly delete any of those pre-existing ports or OVN metadata ports. This unintended port deletion, results in a loss of health monitor capacity, or communication loss with the affected instances.

Workaround: Manually update the 'device_owner' field on existing Load-balancing service health monitor ports to the value of ovn-lb-hm:distributed. Doing so ensures that if the neutron-db-sync-tool is launched, the health monitor or OVN metadata ports are not adversely impacted.

If your deployment has an external Ceph cluster with multiple file systems, you can not create a Shared File System service (Manila) share as expected.

The cephfs_filesystem_name driver configuration parameter that is needed to avoid this situation cannot be set using director’s heat template parameters.

Workaround: Set the "cephfs_filesystem_name" parameter to specify the filesystem that the Shared File System service (Manila) must use via "ExtraConfig".

Add the parameter to an environment file as shown in the following example:

$ cat /home/stack/manila_cephfs_customization.yaml
parameter_defaults:
  ExtraConfig:
    manila::config::manila_config:
      cephfs/cephfs_filesystem_name:
        value: <filesystem>

Replace the value of <filesystem> with the appropriate name and include this environment file with the openstack overcloud deploy command.

A director bug can disrupt or crash client workloads during updates or upgrades to any RHOSP 17.1 version. This bug affects deployments that enable the RHOSP Shared File Systems service (manila) with the CephFS-via-NFS backend.

The bug causes deletion of Ceph NFS export information during update or upgrade operations. This export information is created by the Shared File System Service (manila) when users set up "access rules" on their shares.

When the NFS server goes into a recovery mode, Client workloads can hang and eventually crash if they were actively reading or writing to NFS shares.

Workaround: See Manila shares with Red Hat OpenStack 17.1 can be abruptly disconnected due to export information loss.

This update fixes a bug that negatively affected OVN database operation after the replacement of a bootstrap Controller node. Before this update, you could not use the original bootstrap Controller node hostname and IP address for the replacement Controller node, because the name reuse caused issues with OVN database RAFT clusters.

Now, you can use the original hostname and IP address for the replacement Controller node.

Before this update, in RHOSP 17.1 environments that used the Load-balancing service (octavia) with the OVN service provider driver, load balancer health checks for floating IP addresses (FIPs) were not properly populated with the protocol port. Requests to the FIPs were incorrectly distributed to load balancer members that were in the `ERROR`state.

With this update, the issue is resolved, and any new load balancer health checks for floating IP addresses (FIPs) are properly populated with the protocol port. If you created health monitors before deploying this update, you must recreate them to resolve the port issue.

In RHOSP 17.1.1, the tripleo_frr_bgp_peers role-specific parameter can now be used to specify a list of IP addresses or hostnames for Free Range Routing (FRR) to peer with.

Example

  ControllerRack1ExtraGroupVars:
    tripleo_frr_bgp_peers: ["172.16.0.1", "172.16.0.2"]

This release includes a Technology Preview of the optional feature that you can use to define custom router flavors and create routers with the custom router flavors.

For more information, see Creating custom virtual routers with router flavors.

If you use IPv6 to connect to instances during migration to the OVN mechanism driver, connection to the instances might be disrupted for up to several minutes when the ML2/OVS services are stopped.

The router advertisement daemon radvd for IPv6 is stopped during migration to the OVN mechanism driver. While radvd is stopped, router advertisements are no longer broadcast. This broadcast interruption results in instance connection loss over IPv6. IPv6 communication is automatically restored once the new ML2/OVN services start.

Workaround: To avoid the potential disruption, use IPv4 instead.

In RHOSP 17.1.1, director does not allow for automatically configuring NS records to match a parent’s NS records. Workaround: Until an automated workaround is provided in a future release, administrators can manually change the Orchestration service (heat) template file that resides on the undercloud in /usr/share/ansible/roles/designate_bind_pool/templates/. In the Jinja template, pools.yaml.j2, remove the code following the line containing ns_records until the next empty line (lines 13-16) and insert appropriate values for their infrastructure. Finally, administrators should redeploy the overcloud.

Example

  ns_records:
    - hostname: ns1.desiexample.com
      priority: 1
    - hostname: ns2.desiexample.com
      priority: 2

In RHOSP 17.1 environments that use BGP dynamic routing, there is currently a known issue where floating IP (FIP) port forwarding fails.

When FIP port forwarding is configured, packets sent to a specific destination port with a destination IP that equals the FIP are redirected to an internal IP from a RHOSP Networking service (neutron) port. This occurs regardless of the protocol that is used: TCP, UDP, and so on.

When BGP dynamic routing is configured, the routes to the FIPs used to perform FIP port forwarding are not exposed, and these packets cannot reach their final destinations.

Currently, there is no workaround.

In RHOSP 17.1.1, there is a known issue during a new deployment where the RHOSP Identity service (keystone) is often not available when the agent-notification service is initializing. This prevents ceilometer from discovering the gnocchi endpoint. As a result, metrics are not sent to gnocchi.

Workaround: Restart the agent-notification service on the Controller node:

$ sudo systemctl restart tripleo_ceilometer_agent_notification.service

The Pacemaker-controlled ceph-nfs resource requires a runtime directory to store some process data. The directory is created when you install or upgrade RHOSP. Currently, a reboot of the Controller nodes removes the directory, and the ceph-nfs service does not recover when the Controller nodes are rebooted. If all Controller nodes are rebooted, the ceph-nfs service fails permanently.

Workaround: If you reboot a Controller node, log into the Controller node and create a /var/run/ceph directory:

$ mkdir -p /var/run/ceph

Repeat this step on all Controller nodes that have been rebooted. If the ceph-nfs-pacemaker service has been marked as failed, after creating the directory, execute the following command from any of the Controller nodes:

$ pcs resource cleanup

Currently, rsyslog stops sending logs to Elasticsearch when Logrotate archives all log files once a day.

Workaround: Add "RsyslogReopenOnTruncate: true" to your environment file during deployment so that Rsyslog reopens all log files on log rotation.

Currently, RHOSP 17.1 is shipped with puppet-rsyslog module, which causes Director to configure rsyslog incorrectly.

Workaround: Manually apply patch [1] in /usr/share/openstack-tripleo-heat-templates/deployment/logging/rsyslog-container-puppet.yaml before deployment to configure Rsyslog correctly.

[1] https://github.com/openstack/tripleo-heat-templates/commit/ce0e3a9a94a4fce84dd70b6098867db1c86477fb

In RHOSP environments with ML2/OVN or ML2/OVS that have DVR enabled and use VLAN tenant networks, east/west traffic between instances connected to different tenant networks is flooded to the fabric.

As a result, packets between those instances reach not only the Compute nodes where those instances run, but also any other overcloud node.

This could cause an impact on the network and it could be a security risk because the fabric sends traffic everywhere.

This bug will be fixed in a later FDP release. You do not need to perform a RHOSP update to obtain the FDP fix.

Currently, the Retbleed vulnerability mitigation in RHEL 9.2 can cause a performance drop for Open vSwitch with Data Plane Development Kit (OVS-DPDK) on Intel Skylake CPUs.

This performance regression happens only if C-states are disabled in the BIOS, hyper-threading is enabled, and OVS-DPDK is using only one hyper-thread of a given core.

Workaround: Assign both hyper-threads of a core to OVS-DPDK or to SRIOV guests that have DPDK running as recommended in the NFV configuration guide.

The logging queue that buffers excess security group log entries sometimes stops accepting entries before the specified limit is reached. As a workaround, you can set the queue length higher than the number of entries you want it to hold.

You can set the maximum number of log entries per second with the parameter NeutronOVNLoggingRateLimit. If the log entry creation exceeds that rate, the excess is buffered in a queue up to the number of log entries that you specify in NeutronOVNLoggingBurstLimit.

The issue is especially evident in the first second of a burst. In longer bursts, such as 60 seconds, the rate limit is more influential and compensates for burst limit inaccuracy. Thus, the issue has the greatest proportional effect in short bursts.

Workaround: Set NeutronOVNLoggingBurstLimit at a higher value than the target value. Observe and adjust as needed.

TCP health monitors in UDP pools might not work as expected, depending on the port number that is used by the monitor. Also the status of the pool members and the health monitors are not correct. This is caused by SELinux rules that break the use of TCP health monitors on specific port numbers in UDP pools.

Workaround (if any): Currently, there is no workaround.

RHOSP 17.1 with the OVN mechanism driver does not support logging of flow events per port or the use of the --target option of the network log create command.

RHOSP 17.1 supports logging of flow events per security groups, using the --resource option of the network log create command. See "Logging security group actions" in Networking with RHOSP.

In RHOSP 17.1 GA, the DNS service (designate) is misconfigured when secure role-based access control (sRBAC) is enabled. The current sRBAC policies contain incorrect rules for designate and must be corrected for designate to function correctly. A possible workaround is to apply the following patch on the undercloud server and redeploy the overcloud:

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/888159

In RHOSP 17.1, there is a known issue of transient packet loss where hardware interrupt requests (IRQs) are causing non-voluntary context switches on OVS-DPDK PMD threads or in guests running DPDK applications.

This issue is the result of provisioning large numbers of VFs during deployment. VFs need IRQs, each of which must be bound to a physical CPU. When there are not enough housekeeping CPUs to handle the capacity of IRQs, irqbalance fails to bind all of them and the IRQs overspill on isolated CPUs.

Workaround: You can try one or more of these actions:

In RHOSP 17.1 that run the DNS service (designate), there is a known issue where the bind9 and unbound services are not restarted if the configuration changes.

Workaround: Manually restart the containers by running the following commands on each controller:

$ sudo systemctl restart tripleo_designate_backend_bind9
$ sudo systemctl restart tripleo_unbound

In RHOSP 17.1.1 environments that use IPv6 networks that run the RHOSP DNS service (designate), the BIND 9 back end server can reject DNS notify messages. This issue is caused because there are often multiple IP addresses for the same network on the same interface, and it can appear that the messages are emanating from sources other than the designate Worker services.

Workaround: Apply the following patches:

Currently, there is no support for Multi-RHEL for the following deployment architectures:

There is a known issue when performing an in-place upgrade from RHOSP 16.2 to 17.1 GA. The collection agent, collectd-sensubility fails to run on RHEL 8 Compute nodes.

Workaround: On affected nodes edit the file, /var/lib/container-config-scripts/collectd_check_health.py, and replace "healthy: .State.Health.Status}" with "healthy: .State.Healthcheck.Status}"/ on line 26.

In RHOSP 17.1 GA environments that use the ML2/OVN mechanism driver, there is a known issue with floating IP port forwarding not working correctly. This problem is caused because VLAN and flat networks distribute north-south network traffic when FIPs are used, and, instead, FIP port forwarding should be centralized on the Controller or the Networker nodes.

Workaround: To resolve this problem and force FIP port forwarding through the centralized gateway node, either set the RHOSP Orchestration service (heat) parameter NeutronEnableDVR to false, or use Geneve instead of VLAN or flat project networks.

In this release of RHOSP, there is a known issue where SR-IOV interfaces that use Intel X710 and E810 series controller virtual functions (VFs) with the iavf driver can experience network connectivity issues that involve link status flapping. The affected guest kernel versions are:

The metadata service can become unavailable after the metadata agent fails in multiple attempts to start a malfunctioning HAProxy child container. The metadata agent logs an error message similar to: `ProcessExecutionError: Exit code: 125; Stdin: ; Stdout: Starting a new child container neutron-haproxy-ovnmeta-<uuid>”.

Workaround: Run podman kill <_container name_> to stop the problematic haproxy child container.

The OVNAvailabilityZone Role parameter is not recognized as expected, which causes availability zone configuration to fail in OVN.

Workaround: Use the OVNCMSOptions parameter to configure OVN availability zones. For example:

ControllerParameters:
  OVNCMSOptions: 'enable-chassis-as-gw,availability-zones=az1'

In RHOSP 17.1 environments that use dynamic routing, updating to RHOSP 17.1.1 does not work properly. Specifically, Free Range Routing (FRR) components are not updated.

Workaround: Apply the following patches on the undercloud before updating RHOSP 17.1:

In RHOSP 17.1.1 environments that use the Load-balancing service (octavia) with the OVN provider driver with a health monitor, the pool load-balancing status incorrectly displays fake members as ONLINE. If no health monitor is being used, then the status fake member displays a normal operation of NO_MONITOR.

Fake load-balancing pool members can occur when a member is not valid, such as when there is a typographical error in the member’s IP address. Health monitors configured for the pool perform no health checks on the fake member, and the global operating status incorrectly considers the fake member as ONLINE when it calculates the pool’s status. Furthermore, if all other members in a pool are in ERROR operating status, an incorrect DEGRADED operating status is assigned to the pool instead of ERROR because a member of the pool is a fake member with an incorrect ONLINE status.

Workaround: Currently, there are no workarounds for this issue.

The Networking service (neutron) does not prevent you from disabling or removing a networking profile, even if that profile is part of a flavor that is in use by a router. The disablement or removal of the profile can disrupt proper operation of the router.

Workaround: Before you disable or remove a networking profile, ensure that it is not part of a flavor that is currently used by a router.

Before this update, a bare-metal overcloud node was listed as active by the metalsmith tool even after being deleted. This happened in environments where the node naming scheme overlapped with the overcloud role naming scheme, which could result in the wrong node being unprovisioned during undeploy. Because the metalsmith tool uses the allocation name (hostname) first to lookup the status of bare-metal nodes, it was sometimes finding deleted nodes as still active.

With this update, nodes to be unprovisioned are now referenced by allocation name (hostname), which ensures that the correct node is always unprovisioned. The nodes are only referenced by node name if the hostname doesn’t exist.

Before this update, a live migration might fail because the database did not update with the destination host details.

With this update, the instance host value in the database is set to the destination host during live migration.

Before this update, Open Virtual Network (OVN) load balancer health checks were not performed if you used Floating IPs (FIP) associated with the Load Balancer Virtual IP (VIP), and traffic was redirected to members in the Error state if the FIP was used.

With this update, if you use Floating IPs (FIP) associated with the Load Balancer Virtual IP (VIP), there is a new load balancer health check created for the FIP, and traffic is not redirected to members in the Error state.

Before this update, the cephadm-ansible logs in /home/stack/config-download/overcloud/cephadm were not rotated. The cephadm_command.log was appended for every overcloud deployment and increased in size. Also, for every openstack overcloud ceph spec operation, the log /home/stack/ansible.log was not rotated.

Now, dated logs are generated for every overcloud deployment, and every Ceph spec operation in the following format:

Before this update, the Compute service (nova) did not confidence-check the content of Virtual Machine Disk (VMDK) image files. By using a specially crafted VMDK image, it was possible to expose sensitive files on the host file system to guests booted with that VMDK image. With this update, the Compute service confidence checks VMDK files and forbids VMDK features that the leak behavior depends on. It is no longer possible to leak sensitive host file system contents using specially crafted VMDK files. This bug fix addresses CVE-2022-47951.

Before this update, the default value of rgw_max_attr_size was 256, which created issues for OpenShift on OpenStack when uploading large images. With this update, the default value of rgw_max_attr_size is 1024.

You can change the value by adding the following configuration to an environment file that you include in your overcloud deployment:

parameters_default:
  CephConfigOverrides:
    rgw_max_attr_size: <new value>

Before this update, a regression in the network configuration for OVS interfaces negatively impacted network performance. With this update, the os-vif OVS plugin has been enhanced to improve network performance on the OVS interfaces of non-Windows instances.

Before this update, in RHOSP 17.1 environments that use RHOSP dynamic routing, there was a known issue where the default value of the Autonomous System Number (ASN) used by the OVN BGP agent differed from the ASN used by FRRouting (FRR).

In 17.1 GA, this issue is resolved. The FrrOvnBgpAgentAsn and FrrBgpAsn default values are valid and can be used without needing to modify them.

Before this release, NovaHWMachineType could not be changed for the lifetime of a RHOSP deployment because the machine type of instances without a hw_machine_type image property would use the newly configured machine types after a hard reboot or migration. Changing the underlying machine type for an instance could break the internal ABI of the instance.

With this release, when launching an instance the Compute service records the instance machine type within the system metadata of the instance. Therefore, it is now possible to change the NovaHWMachineType during the lifetime of a RHOSP deployment without affecting the machine type of existing instances.

This update introduces the security group logging feature. To monitor traffic flows and attempts into and out of an instance, you can configure the Networking Service packet logging for security groups.

You can associate any instance port with one or more security groups and define one or more rules for each security group. For instance, you can create a rule to drop inbound ssh traffic to any instance in the finance security group. You can create another rule to allow instances in that group to send and respond to ICMP (ping) messages.

Then you can configure packet logging to record combinations of accepted and dropped packet flows.

You can use security group logging for both stateful and stateless security groups.

Logged events are stored on the Compute nodes that host the instances, in the file /var/log/containers/stdouts/ovn_controller.log.

This enhancement helps cloud users determine if the reason they are unable to access an "ACTIVE" instance is because the Compute node that hosts the instance is unreachable. RHOSP administrators can now configure the following parameters to enable a custom policy that provides a status in the host_status field to cloud users when they run the openstack show server details command, if the host Compute node is unreachable:

With this update, you can configure your RHOSP deployment to count the quota usage of cores and RAM by querying placement for resource usage and instances from instance mappings in the API database, instead of counting resources from separate cell databases. This makes quota usage counting resilient to temporary cell outages or poor cell performance in a multi-cell environment.

Set the following configuration option to count quota usage from placement:

parameter_defaults:
  ControllerExtraConfig:
    nova::config::nova_config:
      quota/count_usage_from_placement:
        value: 'True'

With this update, you can use the validation framework in the workflow of backup and restore procedures to verify the status of the restored system. The following validations are included:

This enhancement allows cloud administrators to specify an Availability Zone (AZ) by storage back end through director when configuring the Shared File Systems service (manila) back-end storage.

With this update, administrators can use an AZ annotation to logically separate storage provisioning requests and to denote failure domains. AZs configured by administrators are exposed by the Shared File Systems service to end users. End users can request that their workloads be scheduled to specific AZs based on their needs. When configuring multiple storage back ends, administrators might want to tag each back end to different AZs as opposed to denoting a single AZ for all back ends.

Director has new options to denote the storage AZs. Each option corresponds to a supported storage back-end driver. For more information about AZs, see Configuring persistent storage.

With this update, you can configure fence_watchdog that uses sbd, like other fencing devices via tripleo, by defining the respective fencing resource:

parameter_defaults:
  EnableFencing: true
  FencingConfig:
    devices:
    - agent: fence_watchdog
      host_mac: 52:54:00:74:f7:51

As an operator, you must enable sbd and set the watchdog timeout:

parameter_defaults:
  ExtraConfig:
    pacemaker::corosync::enable_sbd: true
    tripleo::fencing::watchdog_timeout: 20

With this enhancement, the LVM volumes installed by the overcloud-hardened-uefi-full.qcow2 whole disk overcloud image are now backed by a thin pool. The volumes are still grown to consume the available physical storage, but are not over-provisioned by default.

The benefits of thin-provisioned logical volumes:

This update introduces the script neutron-remove-duplicated-port-bindings to fix an issue that sometimes affected the handling of failed live migrations.

If a live migration fails, the Compute service (Nova) reverts the migration. The migration reversal implies deleting any object created in the database or in the destination compute node.

However, in some cases after the reversal of a failed live migration, ports were left with duplicate port bindings.

The neutron-remove-duplicated-port-bindings script finds duplicate port bindings and deletes the inactive bindings. You can run the script if a failed live migration results in duplicate port bindings.

With this enhancement, operators can enable the run_arping feature for Pacemaker-managed virtual IPs (VIPs), so that the cluster preemptively checks for duplicate IPs.

To do this, you must add the following configuration to the environment file:

  ExtraConfig:
    pacemaker::resource::ip::run_arping: true

If a duplicate is found, the following error is logged in the /var/log/pacemaker/pacemaker.log file:

Sep 07 05:54:54  IPaddr2(ip-172.17.3.115)[209771]:    ERROR: IPv4 address collision 172.17.3.115 [DAD]
Sep 07 05:54:54  IPaddr2(ip-172.17.3.115)[209771]:    ERROR: Failed to add 172.17.3.115

With this update, you can add project and user name fields to outgoing data collection service (ceilometer) metrics. Previously, cloud administrators had to rely on UUIDs of projects and users to identify tenants. Now you can view a list of projects and user names, not UUIDs.

This enhancement allows users to upgrade from RHOSP 16.2 to RHOSP 17.1 and keep the Red Hat Enterprise Linux (RHEL) 8 based operating systems on the Compute nodes, in combination with nodes running RHEL 9.

Control plane nodes and Storage nodes must be upgraded. The default behavior is that all nodes are upgraded to RHEL 9 unless explicitly configured otherwise.

In RHOSP networking environments, when creating a VM instance, do not bind the instance to a virtual port (vport). Instead, use a port whose IP address is not a member of another port’s allowed address pair.

Binding a vport to an instance prevents the instance from spawning and produces an error message similar to the following:

WARNING nova.virt.libvirt.driver [req-XXXX - - - default default] [instance: XXXXXXXXX] Timeout waiting for [('network-vif-plugged', 'XXXXXXXXXX')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds

If you use IPv6 to connect to instances during migration to the OVN mechanism driver, connection to the instances might be disrupted for up to several minutes when the ML2/OVS services are stopped. To avoid this, use IPv4 instead.

The router advertisement daemon radvd for IPv6 is stopped during migration to the OVN mechanism driver. While radvd is stopped, router advertisements are no longer broadcast. This broadcast interruption results in instance connection loss over IPv6. IPv6 communication is automatically restored once the new ML2/OVN services start.

To avoid the potential disruption, use IPv4 instead.

Currently, in ML2/OVS deployments, Open vSwitch (OVS) does not support offloading OpenFlow rules that have the skb_priority, skb_mark, or output queue fields set. These fields are required for Quality of Service (QoS) support for virtio ports.

If you set a minimum bandwidth rule for a virtio port, the Networking service (neutron) OVS agent marks the traffic of this port with a Packet Mark field. This traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means that no traffic can be offloaded.

Workaround: If your environment includes OVS hardware offload ports, disable packet marking in the nodes that require hardware offloading. When you disable packet marking, it is not possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules are still available.

In the configuration file, set the disable_packet_marking flag to true. When you edit the configuration file, you must restart the neutron_ovs_agent container. For example:

$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
  [ovs]
  disable_packet_marking=True

In RHOSP 17.1, when the DNS service (designate) is deployed, Networking service (neutron) ports created on the undercloud are not deleted when the overcloud is deleted. These ports do not cause operational problems when the overcloud is recreated with or without the DNS service.

Workaround: After the overcloud has been deleted, manually remove the ports by using the openstack port delete command.

In RHOSP 17.1 environments that use BGP dynamic routing, there is currently a known issue where floating IP (FIP) port forwarding fails.

When FIP port forwarding is configured, packets sent to a specific destination port with a destination IP that equals the FIP are redirected to an internal IP from a RHOSP Networking service (neutron) port. This occurs regardless of the protocol that is used: TCP, UDP, and so on.

When BGP dynamic routing is configured, the routes to the FIPs used to perform FIP port forwarding are not exposed, and these packets cannot reach their final destinations.

Currently, there is no workaround.

The Pacemaker-controlled ceph-nfs resource requires a runtime directory to store some process data. The directory is created when you install or upgrade RHOSP. Currently, a reboot of the Controller nodes removes the directory, and the ceph-nfs service does not recover when the Controller nodes are rebooted. If all Controller nodes are rebooted, the ceph-nfs service fails permanently.

Workaround: If you reboot a Controller node, log into the Controller node and create a /var/run/ceph directory:

$ mkdir -p /var/run/ceph

Repeat this step on all Controller nodes that have been rebooted. If the ceph-nfs-pacemaker service has been marked as failed, after creating the directory, execute the following command from any of the Controller nodes:

$ pcs resource cleanup

Currently, Logrotate archives all log files once a day and Rsyslog stops sending logs to Elasticsearch Workaround: Add "RsyslogReopenOnTruncate: true" to your environment file during deployment so that Rsyslog reopens all log files on log rotation.

Currently, RHOSP 17.1 uses an older puppet-rsyslog module with an incorrectly configured Rsyslog. Workaround: Manually apply patch [1] in /usr/share/openstack-tripleo-heat-templates/deployment/logging/rsyslog-container-puppet.yaml before deployment to configure Rsyslog correctly.

There is currently a known issue with guest instances that use Mellanox ConnectX-5, ConnectX-6, and Bluefield-2 NICs with offload (switchdev) ports. It takes a long time to initialize the system when you reboot the operating system from the guest directly, for example, by using the command sudo systemctl reboot --reboot-arg=now. If the instance is configured with two Virtual Functions (VFs) from the same Physical Function (PF), the initialization of one of the VFs might fail and cause a longer initialization time.

Workaround: Reboot the guest instance in a timely manner by using the OpenStack API instead of rebooting the guest instance directly.

Overcloud node provisioning may fail for NFV deployments on some AMD platforms in UEFI boot mode on RHOSP 17.1, when using the following BIOS configuration:

In RHOSP environments with ML2/OVN or ML2/OVS that have DVR enabled and use VLAN tenant networks, east/west traffic between instances connected to different tenant networks is flooded to the fabric.

As a result, packets between those instances reach not only the Compute nodes where those instances run, but also any other overcloud node.

This could cause an impact on the network and it could be a security risk because the fabric sends traffic everywhere.

This bug will be fixed in a later FDP release. You do not need to perform a RHOSP update to obtain the FDP fix.

The Dashboard service (horizon) is currently configured to validate client TLS certificates by default, which breaks the Dashboard service on all TLS everywhere (TLS-e) deployments.

Workaround:

Currently, the Retbleed vulnerability mitigation in RHEL 9.2 can cause a performance drop for Open vSwitch with Data Plane Development Kit (OVS-DPDK) on Intel Skylake CPUs.

This performance regression happens only if C-states are disabled in the BIOS, hyper-threading is enabled, and OVS-DPDK is using only one hyper-thread of a given core.

Workaround: Assign both hyper-threads of a core to OVS-DPDK or to SRIOV guests that have DPDK running as recommended in the NFV configuration guide.

The logging queue that buffers excess security group log entries sometimes stops accepting entries before the specified limit is reached. As a workaround, you can set the queue length higher than the number of entries you want it to hold.

You can set the maximum number of log entries per second with the parameter NeutronOVNLoggingRateLimit. If the log entry creation exceeds that rate, the excess is buffered in a queue up to the number of log entries that you specify in NeutronOVNLoggingBurstLimit.

The issue is especially evident in the first second of a burst. In longer bursts, such as 60 seconds, the rate limit is more influential and compensates for burst limit inaccuracy. Thus, the issue has the greatest proportional effect in short bursts.

Workaround: Set NeutronOVNLoggingBurstLimit at a higher value than the target value. Observe and adjust as needed.

RHOSP 17.1 with the OVN mechanism driver does not support logging of flow events per port or the use of the --target option of the network log create command.

RHOSP 17.1 supports logging of flow events per security groups, using the --resource option of the network log create command. See "Logging security group actions" in Configuring Red Hat OpenStack Platform networking.

In RHOSP 17.1 GA, the DNS service (designate) is misconfigured when secure role-based access control (sRBAC) is enabled. The current sRBAC policies contain incorrect rules for designate and must be corrected for designate to function correctly.

Workaround: Apply the following patch on the undercloud server and redeploy the overcloud:

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/888159

In RHOSP 17.1, there is a known issue of transient packet loss where hardware interrupt requests (IRQs) are causing non-voluntary context switches on OVS-DPDK PMD threads or in guests running DPDK applications.

This issue is the result of provisioning large numbers of VFs during deployment. VFs need IRQs, each of which must be bound to a physical CPU. When there are not enough housekeeping CPUs to handle the capacity of IRQs, irqbalance fails to bind all of them and the IRQs overspill on isolated CPUs.

Workaround: You can try one or more of these actions:

Currently, when a bootstrap Controller node is replaced, the OVN database cluster is partitioned: with two database clusters for both the northbound and southbound databases. This situation makes instances unusable.

To find the name of the bootstrap Controller node, run the following command:

ssh tripleo-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

Workaround: Perform the steps described in Red Hat KCS solution 7024434: Recover from partitioned clustered OVN database.

Currently, there is no support for Multi-RHEL for the following deployment architectures:

There is a known issue when performing an in-place upgrade from RHOSP 16.2 to 17.1 GA. The collection agent, collectd-sensubility fails to run on RHEL 8 Compute nodes.

Workaround: On affected nodes edit the file, /var/lib/container-config-scripts/collectd_check_health.py, and replace "healthy: .State.Health.Status}" with "healthy: .State.Healthcheck.Status}"/ on line 26.

In RHOSP 17.1 GA environments that use the ML2/OVN mechanism driver, there is a known issue with floating IP port forwarding not working correctly. This problem is caused because VLAN and flat networks distribute north-south network traffic when FIPs are used, and, instead, FIP port forwarding should be centralized on the Controller or the Networker nodes.

Workaround: To resolve this problem and force FIP port forwarding through the centralized gateway node, either set the RHOSP Orchestration service (heat) parameter NeutronEnableDVR to false, or use Geneve instead of VLAN or flat project networks.

In this release of RHOSP, there is a known issue where SR-IOV interfaces that use Intel X710 and E810 series controller virtual functions (VFs) with the iavf driver can experience network connectivity issues that involve link status flapping. The affected guest kernel versions are:

There is currently a known issue when using a Red Hat Ceph Storage (RHCS) back end for volumes that can prevent instances from being rebooted, and may lead to data corruption. This occurs when all of the following conditions are met:

Workaround: Do not retype in-use volumes that meet all of these listed conditions.

The metadata service can become unavailable after the metadata agent fails in multiple attempts to start a malfunctioning HAProxy child container. The metadata agent logs an error message similar to: `ProcessExecutionError: Exit code: 125; Stdin: ; Stdout: Starting a new child container neutron-haproxy-ovnmeta-<uuid>”.

Workaround: Run podman kill <_container name_> to stop the problematic haproxy child container.

If you download RHOSP 17.1.0 GA in the first few days of its availability, you might find that the version description in the file /etc/rhosp/release incorrectly includes the Beta designation, as shown in the following example.

(overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release
Red Hat OpenStack Platform release
17.1.0 Beta (Wallaby)

Workaround: If your GA deployment is affected, run the following command: # dnf -y update rhosp-release

If you download RHOSP 17.1.0 GA in the first few days of its availability, you might find that the version description in the file /etc/rhosp/release incorrectly includes the Beta designation, as shown in the following example.

(overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release
Red Hat OpenStack Platform release
17.1.0 Beta (Ussri)

Workaround: If your GA deployment is affected, run the following command: # dnf -y update rhosp-release

The ML2/OVS mechanism driver is deprecated since RHOSP 17.0.

Over several releases, Red Hat is replacing ML2/OVS with ML2/OVN. For instance, starting with RHOSP 15, ML2/OVN became the default mechanism driver.

Support is available for the deprecated ML2/OVS mechanism driver through the RHOSP 17 releases. During this time, the ML2/OVS driver remains in maintenance mode, receiving bug fixes and normal support, and most new feature development happens in the ML2/OVN mechanism driver.

In RHOSP 18.0, Red Hat plans to completely remove the ML2/OVS mechanism driver and stop supporting it.

If your existing RHOSP deployment uses the ML2/OVS mechanism driver, start now to evaluate a plan to migrate to the mechanism driver. Migration is supported in RHOSP 16.2 and 17.1.

Red Hat requires that you file a proactive support case before attempting a migration from ML2/OVS to ML2/OVN. Red Hat does not support migrations without the proactive support case. See How to open a proactive case for a planned activity on Red Hat OpenStack Platform?.

Monitoring of API health status via podman using sensubility is deprecated in RHOSP 17.1.

Only the sensubility layer is deprecated. API health checks remain in support. The sensubility layer exists for interfacing with Sensu, which is no longer a supported interface.

Before this update, a live migration might fail because the database did not update with the destination host details.

With this update, the instance host value in the database is set to the destination host during live migration.

Before this update Open Virtual Network (OVN) load balancer health checks were not performed if you used Floating IPs (FIP) associated to the Load Balancer Virtual IP (VIP), and traffic was redirected to members in the Error state if the FIP was used.

With this update, if you use Floating IPs (FIP) is associated to the Load Balancer Virtual IP (VIP), there is a new load balancer health check created for the FIP, and traffic is not redirected to members in the Error state.

Before this update, the Compute service (nova) did not confidence-check the content of Virtual Machine Disk (VMDK) image files. By using a specially crafted VMDK image, it was possible to expose sensitive files on the host file system to guests booted with that VMDK image. With this update, the Compute service confidence checks VMDK files and forbids VMDK features that the leak behavior depends on. It is no longer possible to leak sensitive host file system contents using specially crafted VMDK files.

Before this update, the default value of rgw_max_attr_size was 256, which created issues for OpenShift on OpenStack when uploading large images. With this update, the default value of rgw_max_attr_size is 1024.

You can change the value by adding the following configuration to an environment file that you include in your overcloud deployment:

parameters_default:
  CephConfigOverrides:
    rgw_max_attr_size: <new value>

Before this release, NovaHWMachineType could not be changed for the lifetime of a RHOSP deployment because the machine type of instances without a hw_machine_type image property would use the newly configured machine types after a hard reboot or migration. Changing the underlying machine type for an instance could break the internal ABI of the instance.

With this release, when launching an instance the Compute service records the instance machine type within the system metadata of the instance. Therefore, it is now possible to change the NovaHWMachineType during the lifetime of a RHOSP deployment without affecting the machine type of existing instances.

This update introduces the security group logging feature. To monitor traffic flows and attempts into and out of a virtual machine instance, you can configure the Networking Service packet logging for security groups.

You can associate any virtual machine instance port with one or more security groups and define one or more rules for each security group. For instance, you can create a rule to drop inbound ssh traffic to any virtual machine in the finance security group. You can create another rule to allow virtual machines in that group to send and respond to ICMP (ping) messages.

Then you can configure packet logging to record combinations of accepted and dropped packet flows.

You can use security group logging for both stateful and stateless security groups.

Logged events are stored on the compute nodes that host the virtual machine instances, in the file /var/log/containers/stdouts/ovn_controller.log.

This enhancement helps cloud users determine if the reason they are unable to access an "ACTIVE" instance is because the Compute node that hosts the instance is unreachable. RHOSP administrators can now configure the following parameters to enable a custom policy that provides a status in the host_status field to cloud users when they run the openstack show server details command, if the host Compute node is unreachable:

With this update, you can use the validation framework in the workflow of backup and restore procedures to verify the status of the restored system. The following validations are included:

With this enhancement, the LVM volumes installed by the overcloud-hardened-uefi-full.qcow2 whole disk overcloud image are now backed by a thin pool. The volumes are still grown to consume the available physical storage, but are not over-provisioned by default.

The benefits of thin-provisioned logical volumes:

In RHOSP 17.1, Red Hat recommends that all physical functions (PFs) on the same NIC hardware use drivers that are in the same space. PFs on the same NIC should all use drivers that run in either the user space or in the kernel space.

For example, if PF1 on NIC1 is used by the DPDK PMD driver, then PF2 on NIC1 should not use the kernel driver. In this example, the PFs on NIC1 should both use the DPDK PMD driver or both use the kernel driver.

If you use IPv6 to connect to VM instances during migration to the OVN mechanism driver, connection to the instances might be disrupted for up to several minutes when the ML2/OVN services start.

The router advertisement daemon radvd for IPv6 is stopped during migration to the OVN mechanism driver. While radvd is stopped, router advertisements are no longer broadcast. This broadcast interruption results in VM instance connection loss over IPv6. IPv6 communication is automatically restored once the new ML2/OVN services start.

Workaround: To avoid the potential disruption, use IPv4 instead.

Currently, in ML2/OVS deployments, Open vSwitch (OVS) does not support offloading OpenFlow rules that have the skb_priority, skb_mark, or output queue fields set. These fields are required for Quality of Service (QoS) support for virtio ports.

If you set a minimum bandwidth rule for a virtio port, the Networking service (neutron) OVS agent marks the traffic of this port with a Packet Mark field. This traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means that no traffic can be offloaded.

Workaround: If your environment includes OVS hardware offload ports, disable packet marking in the nodes that require hardware offloading. When you disable packet marking, it is not possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules are still available.

In the configuration file, set the disable_packet_marking flag to true. When you edit the configuration file, you must restart the neutron_ovs_agent container. For example:

$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
  [ovs]
  disable_packet_marking=True

In RHOSP 17.0, the DNS service (designate) and the Load-balancing service (octavia) are misconfigured for high availability. The RHOSP Orchestration service (heat) templates for these services use the non-Pacemaker version of the Redis template.

Workaround: include environments/ha-redis.yaml in the overcloud deploy command after the enable-designate.yaml and octavia.yaml environment files.

In RHOSP 17.1 environments that use BGP dynamic routing, there is currently a known issue where floating IP (FIP) port forwarding fails.

When FIP port forwarding is configured, packets sent to a specific destination port with a destination IP that equals the FIP are redirected to an internal IP from a RHOSP Networking service (neutron) port. This occurs regardless of the protocol that is used: TCP, UDP, and so on.

When BGP dynamic routing is configured, the routes to the FIPs used to perform FIP port forwarding are not exposed, and these packets cannot reach their final destinations.

Currently, there is no workaround.

Red Hat has not validated the RHOSP 17.1 beta release on NFV deployments with AMD processors. Testing is underway now with plans to validate the application in a future release.

Do not use RHOSP 17.1 NFV deployments with AMD hardware for production until Red Hat validates the application. Any use of this pre-tested application is at risk for unintended results.

In RHOSP 17.1 environments with ML2/OVN, DVR enabled and using VLAN tenant networks, east/west traffic between VMs connected to different tenant networks is flooded to the fabric.

As a result, packets between those VMs reach not only the compute nodes where those VMs run, but also any other overcloud node.

This could cause an impact in the network side and it could be a security risk because the fabric sends traffic everywhere.

This bug will be fixed in a later FDP release, so no RHOSP update is needed to obtain it.

Currently, a known issue in the Ceph RADOS Gateway component in Red Hat Ceph Storage (RHCS) 6.0 causes authorization with Identity service (keystone) tokens to fail. See https://bugzilla.redhat.com/2188266.

As a result, when you configure your deployment with Red Hat Ceph Storage using RADOS Gateway as the object-store server, Object Storage service (swift) clients fail and return code 403/Unauthorized. The issue did not manifest in tests that deployed pre-release versions of RHCS 6.1, which was released for general availability on June 15, 2023.

Also, OpenShift integration on OpenStack has not been validated for beta because the default configuration uses RADOS Gateway. The following workaround is expected to mitigate the issue and enable you to do preliminary tests with OpenShift integration on OpenStack.

Workaround: Deploy the Object Storage service (swift) as the object-store server instead of RADOS Gateway, even when enabling Ceph Storage for persistent Block Storage service (cinder) or Image service (glance) storage and ephemeral Compute service (nova) storage. To do this, replace the cephadm.yaml environment file with the cephadm-rbd-only.yaml in the deployment command line.

When you configure the OpenStack environment with the Object Storage service (swift) instead of RADOS Gateway as the object-store server, Object Storage service (swift) clients work as expected.

In RHOSP 17.1 environments that use BGP dynamic routing with OVN, there is a known issue where the default value of the Autonomous System Number (ASN) used by the OVN BGP agent differs from the ASN used by FRRouting (FRR).

Workaround: ensure that the values for the tripleo parameters used in the undercloud and overcloud configuration, FrrBgpAsn and FrrOvnBgpAgentAsn, are identical.

There is currently a known issue where the Retbleed vulnerability mitigation in RHEL 9.2 can cause a performance drop for Open vSwitch with Data Plane Development Kit (OVS-DPDK) on Intel Skylake CPUs.

This performance regression happens only if C-states are disabled in the BIOS, hyper-threading is enabled, and OVS-DPDK is using only one hyper-thread of a given core.

Workaround: Assign both hyper-threads of a core to OVS-DPDK or to SRIOV guests that have DPDK running as recommended in the NFV configuration guide.

In RHOSP 17.1 environments that use BGP dynamic routing, there is currently a known issue where the OVN BGP agents that are running on overcloud nodes fail because of a bug in a shipped library (pyroute2). When this issue occurs, no new routes are advertised from the affected node, and there might be a loss of connectivity with new or migrated VMs, new load balancers, and so on.

Workaround: Install an updated version of pyroute2 in the ovn_bgp_agent container, by adding the following lines to containers-prepare-parameter.yaml:

ContainerImagePrepare:
- push_destination: true
  ...
  includes:
  - nova-compute
  modify_role: tripleo-modify-image
  modify_append_tag: "-hotfix"
  modify_vars:
    tasks_from: rpm_install.yml
    rpms_path: /home/stack/nova-hotfix-pkgs
  ...

For more information, see Installing additional RPM files to container images.

The logging queue that buffers excess security group log entries sometimes stops accepting entries before the specified limit is reached. As a workaround, you can set the queue length higher than the number of entries you want it to hold.

You can set the maximum number of log entries per second with the parameter NeutronOVNLoggingRateLimit. If the log entry creation exceeds that rate, the excess is buffered in a queue up to the number of log entries that you specify in NeutronOVNLoggingBurstLimit.

The issue is especially evident in the first second of a burst. In longer bursts, such as 60 seconds, the rate limit is more influential and compensates for burst limit inaccuracy. Thus, the issue has the greatest proportional effect in short bursts.

Workaround: Set NeutronOVNLoggingBurstLimit at a higher value than the target value. Observe and adjust as needed.

Currently, DNS-as-a-Service (designate) is misconfigured when secure role-based access control (SRBAC) is enabled. If you configure both SRBAC and DNS-as-a-Service, the RHOSP deployment fails. Workaround: For a successful deployment, apply the following patches on the undercloud server:

The ML2/OVS mechanism driver is deprecated since RHOSP 17.0.

Over several releases, Red Hat is replacing ML2/OVS with ML2/OVN. For instance, starting with RHOSP 15, ML2/OVN became the default mechanism driver.

Support is available for the deprecated ML2/OVS mechanism driver through the RHOSP 17 releases. During this time, the ML2/OVS driver remains in maintenance mode, receiving bug fixes and normal support, and most new feature development happens in the ML2/OVN mechanism driver.

In RHOSP 18.0, Red Hat plans to completely remove the ML2/OVS mechanism driver and stop supporting it.

If your existing RHOSP deployment uses the ML2/OVS mechanism driver, start now to evaluate a plan to migrate to the mechanism driver. Migration is supported in RHOSP 16.2 and 17.1.

Red Hat requires that you file a proactive support case before attempting a migration from ML2/OVS to ML2/OVN. Red Hat does not support migrations without the proactive support case. See How to submit a Proactive Case.

Monitoring of API health status via podman using sensubility is deprecated in RHOSP 17.1.

Only the sensubility layer is deprecated. API health checks remain in support. The sensubility layer exists for interfacing with Sensu, which is no longer a supported interface.

The Derived Parameters feature is removed. The Derived Parameters feature is configured using the --plan-environment-file option of the openstack overcloud deploy command.

Workaround / Migration Instructions

NFV and HCI overclouds require system tuning. There are many different options for system tuning. The Derived Parameters functionality tuned systems with director using to inspect hardware inspection data and set tuning parameters using the --plan-environment-file option of the openstack overcloud deploy command. The Derived Parameters functionality is removed in 17.1.

The following parameters were tuned by this functionality: