Chapter 4. Technical notes

The technology preview support added in RHOSP 16.1 for configuring NVDIMM Compute nodes to provide persistent memory for instances has been deprecated in RHOSP 16.2.5, and will be removed in RHOSP 17.0. Red Hat is removing support for persistent memory from RHOSP 17.0 and future releases in response to the announcement by the Intel Corporation on July 28, 2022 that they are discontinuing investment in their Intel® Optane™ business:

Cloud operators must ensure that no instances use the vPMEM feature before upgrading to 17.1. (BZ#2187380)

This update fixes the following PowerMax Block Storage (cinder) driver issues:

This RHOSP 16.2.4 update makes it possible for you to correct a libvirt version incompatibility before updating to RHOSP 16.2.4. If you have the incompatibility issue and do not addresss it before updating to RHOSP 16.2.4, the update might leave instances in an unmanageable state.

Before updating to 16.2.4, see the KCS article Workaround for a libvirt version-compat issue (bug 2109350) when updating RHOSP 16.2.0.

Perform the steps in the article to determine whether your update path is affected by the libvirt incompatibility issue. If it is affected, perform the steps to resolve the issue. (BZ#2109350)

Options for Layer 3 (L3) agents:

Options for for Open vSwitch (OVS) agents:

Options for ML2/OVN back ends:

This update fixes a bug that prevented the ceilometer-agent-compute service from collecting libvirt-related metrics.

Previously, the libvirt service started after the ceilometer-agent-compute service, which resulted in "Permission denied" failures and loss of metrics data. Now the libvirt service starts before the ceilometer-agent-compute service and the service can properly collect metrics. (BZ#2092088)

Previously, Red Hat Ceph Storage nodes were incorrectly configured to consume OpenStack high availability, advanced-virt, and fast-datapath repos during Leapp upgrades. The previous bug fix for this issue introduced an override that caused role-based parameters to work incorrectly.

With this update, the role-based parameter implementation is fixed and the correct repositories are enabled for Red Hat Ceph Storage nodes. This update fixes the issue in Red Hat OpenStack Platform environments 16.2 and later that use the Red Hat Ceph Storage role. (BZ#2094377)

This update fixes a bug that caused ceilometer-agent-ipmi to write log data inside the container namespace instead of on the host as expected.

The improper placement of the content inside the container increased the container size, prevented proper log rotation, and resulted in loss of the log data when the container was deleted or rebuilt.

Now ceilometer-agent-ipmi writes the logs to the host in /var/log/containers/ceilometer/ as expected. (BZ#2138395)

This update fixes a bug that causes connectivity loss after certain updates to RHOSP 16.2.2 and 16.2.3. If you are planning to update to a RHOSP 16.2 release, update to RHOSP 16.2.4 to avoid connectivity loss.

The bug is triggered by a database schema change in OVN 21.12, which is introduced in RHOSP 16.2.2. and 16.2.3. OVN 21.12 contains a new column that is not present in earlier versions. OVN database schema changes should not cause a problem in OpenStack, but this particular change is affected by a bug.

In particular, instance connectivity is lost for a variable amount of time (from 20 seconds to 3 minutes) when you run the following command:

$ openstack overcloud external-update run --stack overcloud --tags ovn

To avoid the bug, do not update to RHOSP 16.2.2. or 16.2.3. Update to RHOSP 16.2.4 instead. (BZ#2111871)

Before this update, the RHOSP Bare Metal service (ironic) could lose its connection to the remote Redfish baseboard management controller (BMC) resulting in the bare metal node entering a maintenance state and with its power status changing to None. Depending on a site’s environmental factors, some or all of the bare metal nodes could be in this unwanted maintenance state for an extended period of time.

Transient network connectivity issues caused by high packet loss to the BMC caused connection caching issues when using Redfish. In cases where a session token needed to be renegotiated, the cached session object was never invalidated and connectivity was lost to the BMC.

With this update, the Bare Metal service now initializes an entirely new cached session with a remote Redfish BMC when connectivity or authentication issues are detected. Additionally, this allows updated credentials to be leveraged should the bare metal node’s BMC passwords be changed after initial configuration. (BZ#2064767)

Open vSwitch (OVS) does not support offloading OpenFlow rules that have the skb_priority, skb_mark, or output queue fields set. Those fields are needed to provide quality-of-service (QoS) support for virtio ports.

If you set a minimum bandwidth rule for a virtio port, the Neutron Open vSwitch agent marks the traffic of this port with a Packet Mark Field. As a result, this traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means no traffic can be offloaded.

As a workaround, if your environment includes OVS hardware offload ports, disable the packet marking in the nodes that require hardware offloading. After you disable the packet marking, it will not be possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules will still be available.

In the configuration file, set the disable_packet_marking flag to true. After you edit the configuration file, you must restart the neutron_ovs_agent container. For example:

$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
  [ovs]
  disable_packet_marking=True

(BZ#2092946)

Before this update, during the replacement of a DCN node, the etcd service on the replacement node failed to start and caused the cinder-volume service on that node to fail. This failure was caused by the replacement for a DCN node attempting to start the etcd service as if it were bootstrapping a new etcd cluster, instead of joining the existing etcd cluster.

With this update, a new parameter has been added, EtcdInitialClusterState. When EtcdInitialClusterState is set to existing, the DCN node starts etcd correctly, which causes the cinder-volume service to run successfully. (BZ#2055409)

Before this update, attempts to stop the cinder-volume service running in active-active mode resulted in the failed state. The cause for these failures was that the service was not allowing sufficient time to properly stop itself.

With this update, the time period allocated to stopping the service has been extended, and the service moves to the inactive state when you stop it. (BZ#2056918)

In cases where high CPU use was monitored in a multi-core system, the calculation for CPU use was inaccurate.

With this update, the calculation of CPU use in a multi-core scenario is now accurate. The latest STF dashboards have been adjusted to incorporate this update. (BZ#2006970)

This update fixes a bug that caused some virtual machine (VM) bootup operations to fail when multiple VMs were booted simultaneously from an image.

Previously, the Block Storage service (cinder) GPFS SpectrumScale driver did not correctly detect when the storage back end supported copy-on-write (COW) mode. The driver disabled COW features such as the ability to rapidly create volumes from an image. When booting multiple instances simultaneously from an image, some instances timed out when copying the image to its boot volume.

The GPFS Block Storage service driver now properly detects when the storage back end supports COW mode. You can now spawn multiple VM instances simultaneously. (BZ#2022018)

Before this update, when creating a snapshot with PowerMaxOS 5978.711, REST experienced a payload response change and caused the device label to modify its format. The underlying data from the solutions enabler changed and no longer contained a colon character (:). This caused an IndexError exception in the PowerMax Driver:

IndexError: list index out of range

With this update, the problem is resolved in PowerMaxOS 5978.711 and later. (BZ#2003762)

With this update, you can set QoS maximum bandwidth limit, egress direction rules on hardware-offloaded ports in a ML2/OVS deployment. To set the policy, use the normal QoS policy/rules methods.

The back end uses ip link commands to enforce the policy instead of the normal OVS QoS engine, because the OVS meter action cannot be offloaded. See meter action is not offloaded. (BZ#1971545)

This update fixes a bug that caused unintended results when converting Dell EMC PERC H755 RAID controller physical disks to non-RAID mode.

The conversion erroneously created RAID-0 virtual disks and moved them to the Online state, consuming a physical disk.

The RAID-0 virtual disks are no longer created during the conversions. (BZ#2010246)

This update fixes a bug that prevented connections to the Ceph storage backend with Ceph client release 15.2.0 (Octopus) and later, affecting Red Hat Ceph Storage 5.0 and later.

A temporary configuration file generated to enable a Ceph connection did not include a '[global]' section marker. This update adds the '[global]' section marker to the temporary file.

The section marker was introduced in Ceph client release 0.94.0 (Hammer). Starting with the Octopus release, Ceph requires the presence of the marker. This fix is backward compatible to Red Hat Ceph Storage 4.x. (BZ#2024684)

With this technology preview update, you can set the following parameters to configure OVS PMD automatic load balance:

OvsPmdAutoLb: Enable/disable the OVS DPDK PMD Auto Load Balance feature. Values: true or false. OVS DPDK uses the default value of false.

OvsPmdLoadThreshold: Set the minimum PMD thread load threshold for OVS DPDK PMD Auto Load Balance feature. Set a value from 0 to 100 to specify the minimum PMD thread load threshold (% of used cycles) of any non-isolated PMD threads when a PMD Auto Load Balance might be triggered.

OvsPmdImprovementThreshold: Set PMD load variance improvement threshold for OVS DPDK PMD Auto Load Balance feature. Set a value from 0 to 100 to specify the minimum evaluated percentage improvement in load distribution across the non-isolated PMD threads that allows a PMD Auto Load Balance to occur.

OvsPmdRebalInterval: Set PMD auto load balancing interval for OVS DPDK PMD Auto Load Balance feature. Set a value from 0 to 20,000 to specify the minimum time (in minutes) between 2 consecutive PMD Auto Load Balancing iterations. (BZ#1952060)

openstack-tripleo-heat-templates: data leak of internal URL through keystone_authtoken (CVE-2021-4180)

For more details about the security issue(s), including the impact, a CVSS score, and other related information, refer to the CVE page(s) listed in the References section.

Disable the advanced-virt-for-rhel-8 repository before you install Red Hat OpenStack Platform (RHOSP) 16.2, update from RHOSP 16.2 to a newer maintenance release, or upgrade from 16.1 to 16.2.

RHOSP hosts do not require the advanced-virt-for-rhel-8 repository. If you do not disable it, dependency issues cause the installation, update, or upgrade to fail. The dependency failures happen because the advanced-virt-for-rhel-8-x86_64-rpms repository is based on RHEL 8.latest, which does not work with RHEL 8.4.

As a workaround, disable the repositories. Perform the steps appropriate for your installation, update, or upgrade scenario.

Scenario: new 16.2 installation or update from 16.2 to later version of 16.2.

$ subscription-manager repos --disable advanced-virt-for-rhel-8-x86_64-rpms

$ dnf module disable -y virt:av

$ dnf module enable -y virt:rhel

Scenario: upgrade from 16.1→16.2.

$ subscription-manager repos --disable advanced-virt-for-rhel-8-x86_64-rpms

$ dnf module disable -y virt:8.2

$ dnf module enable -y virt:rhel (BZ#2027787)

The Block Storage service (cinder) can now use an external NFS share to perform image format conversion of Image service (glance) images on the overcloud Controller nodes. Using this functionality prevents the space on the node from being completely filled during a conversion operation.

See Configuring an external NFS share for conversion. (BZ#1886762)

The command "leapp answer --section authselect_check.confirm=True" has not been run before triggering the leapp upgrade.

With this update, the leapp process completes successfully without the need to run the command "leapp answer --section authselect_check.confirm=True" before triggering the leapp upgrade command. (BZ#1978228)

This update adds an NTP validation step to deployments. To include the validation step in your deployment, include the argument --ntp-server <ntp_server_name> in the openstack deploy command. Replace <ntp_server_name> with the name of a valid, reachable NTP server.

If director cannot reach the specified NTP server, the deployment fails. This validation prevents synchronization failures from occurring later in the deployment. (BZ#2034189)

In Red Hat OpenStack Platform (RHOSP) deployments that use the Modular Layer 2 plug-in with the Open vSwitch (ML2/OVS) mechanism driver, there is currently a known issue where the Orchestration service (heat) parameter, NeutronL3AgentAvailabilityZone does not set the relevant Neutron L3 agent parameter correctly.

Workaround: use a custom hieradata statement to set this value. In the example that follows, replace [ROLE] with the composable role name that is appropriate for your site.

[ROLE]ExtraConfig:
    neutron::agents::l3::availability_zone: role_availability_zone

For more information, see Puppet: Customizing hieradata for roles in the Advanced Overcloud Customization guide. (BZ#1983748)

This enhancement enables the experimental rsyslog reopenOnTruncate setting to ensure that rsyslog immediately recognizes when a logrotation happens on a file. The setting affects every service configured to work with rsyslog.

When rsyslog reopenOnTruncate is disabled, rsyslog waits for a log file to fill to its original capacity before consuming any additional logs. (BZ#1949675)

Typically, when you create an encrypted volume from a snapshot of an encrypted volume, the source volume is the same size or smaller than the destination volume.

In previous releases, if you created an encrypted volume from a snapshot of an encrypted volume, and the destination volume was close to or equal to the size of the source volume, the Block Storage service (cinder) silently truncated the data in the new destination volume.

With this release, the Block Storage service calculates the size of the destination volume to include the current size of the encryption header, which eliminates the data truncation. (BZ#1772531)

Before this update, the NetApp SolidFire driver would create a duplicate volume when the API response is lost due to a connection error and the driver retries the API request. This occurred when the SolidFire back end successfully received and processed a create volume operation, but failed to deliver the response back to the driver. This update resolves the issue by:

Previously, the Shared File Systems service (manila) API that brings external shares into service management did not check for duplicated export locations. An existing share brought into the service multiple times results in an inconsistent state.

With this release, the API evaluates the export locations of known or existing shares before allowing external shares to be managed, and prevents existing shares from being erroneously brought into the Shared File Systems service again. (BZ#1849843)

Before this update, the Shared File Systems service (manila) dashboard had dynamic form elements whose names could potentially cause the forms to become unresponsive. This meant that the creation of share groups, share networks, and shares within share networks did not work.

With this update, dynamic elements whose names could be problematic are encoded, which means that creation of share groups, share networks, and shares within share networks functions normally. (BZ#1974979)

When an instance is created, the Compute service (nova) sanitizes the instance display name to generate a valid hostname when DNS integration is enabled in the Networking service (neutron).

Before this update, the sanitization did not replace periods ('.') in instance names, for example, 'rhel-8.4'. This could result in display names being recognized as Fully Qualified Domain Names (FQDNs) which produced invalid hostnames. When instance names contained periods and DNS integration was enabled in the Networking service, the Networking service would reject the invalid hostname resulting in a failure to create the instance and a HTTP 500 server error from the Compute service.

With this update, periods are now replaced by hyphens in instance names to prevent hostnames being parsed as FQDNs. You can continue to use free-form strings for instance display names. (BZ#1919855)

With this enhancement, you can improve the performance of live migrations by using the following new parameters:

This update fixes an issue that caused Networking service (neutron) agents, such as Networking service DHCP, to fail when they tried to create resources in OVN. This was caused by residual data left in the OVN databases when QoS rules were created for floating IPs.

This update eliminates the residual data and fixes the problem. (BZ#1978158)

Starting with Red Hat Enterprise Linux (RHEL) version 8.3, support for the Intel Transactional Synchronization Extensions (TSX) feature is disabled by default. Currently, this causes instance live migration to fail when migrating from hosts where the TSX kernel argument is enabled to hosts where the TSX kernel argument is disabled.

This impact applies only to Intel hosts that support the TSX feature. For more information about the CPUs that are affected by this issue, see Affected Configurations.

For more information, review the following Red Hat Knowledgebase solution Guidance on Intel TSX impact on OpenStack guests. (BZ#1975240)

Before this fix, grub2 tooling wrote kernel argument changes to /boot/grub2/grubenv. This file was not available to UEFI boot systems, and caused kernel argument changes not to persist across reboots on UEFI boot nodes.

This fix changes both the /boot/grub2/grubenv file and the /boot/efi/EFI/redhat/grubenv files when you make kernel argument changes. As a result, RHOSP director now applies persistent Kernel argument changes for UEFI boot nodes. (BZ#1987092)

During stack update the KernelArgs could be modified or appended. A reboot of the affected nodes needs to be performed manually.

For example, if the current deployment has the following configuration, it is possible to change hugepages=64, or add or remove arguments during the stack update:

`KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 intel_iommu=on iommu=pt isolcpus=1-11,13-23"

For example:

KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt isolcpus=1-24"
KernelArgs: "isolcpus=1-11,13-23"