Release Notes

This release includes the capability to deploy multiple overclouds from a single undercloud.

With Red Hat OpenStack Platform director, you can now deploy the Block Storage service (cinder) in an active-active configuration if the back end driver supports this configuration. As of GA, only the Ceph RADOS Block Device (RBD) back end driver supports an active-active configuration.

The new cinder-volume-active-active.yaml file defines the active-active cluster name by assigning a value to the CinderVolumeCluster parameter. CinderVolumeCluster is a global Block Storage parameter, and prevents you from including clustered (active-active) and non-clustered back ends in the same deployment.

The cinder-volume-active-active.yaml file causes director to use the non-Pacemaker, cinder-volume Orchestration service template, and adds the etcd service to your Red Hat OpenStack Platform deployment as a distributed lock manager (DLM).

Red Hat OpenStack Platform Bare Metal service (ironic) now has a BIOS management interface, with which you can inspect and modify a device’s BIOS configuration.

In Red Hat OpenStack Platform 16.0, the Bare Metal service supports BIOS management capabilities for data center devices that are Redfish API compliant. The Bare Metal service implements Redfish calls through the Python library, Sushy.

A new Compute (nova) parameter is available, QemuMemoryBackingDir, which specifies the directory in which to store the memory backing file when a libvirt memoryBacking element is configured with source type="file" and access mode="shared".

Note: The memoryBacking element is only available from libvirt 4.0.0 and QEMU 2.6.0.

In Red Hat OpenStack Platform 16.0, you can separate the provisioning and deployment stages of your deployment into distinct steps:

In Red Hat OpenStack Platform 16.0, live migrations with OVN enabled now succeed, as the flag, live_migration_wait_for_vif_plug, is enabled by default.

Previously, live migrations failed, because the system was waiting for OpenStack Networking (neutron) to send vif_plugged notifications.

With this enhancement, support for live migration of instances with attached SR-IOV-based neutron interfaces has been added. Neutron SR-IOV interfaces can be grouped into two categories: direct mode and indirect mode. Direct mode SR-IOV interfaces are directly attached to the guest and exposed to the guest OS. Indirect mode SR-IOV interfaces have a software interface, for example, a macvtap, between the guest and the SR-IOV device. This feature enables transparent live migration for instances with indirect mode SR-IOV devices. As there is no generic way to copy hardware state during a live migration, direct mode migration is not transparent to the guest. For direct mode interfaces, mimic the workflow already in place for suspend and resume. For example, with SR-IOV devices, detach the direct mode interfaces before migration and re-attach them after the migration. As a result, instances with direct mode SR-IOV ports lose network connectivity during a migration unless a bond with a live migratable interface is created within the guest.

Previously, it was not possible to live migrate instances with SR-IOV-based network interfaces. This was problematic as live migration is frequently used for host maintenance and similar actions. Previously, the instance had to be cold migrated which involves downtime for the guest.

This enhancement results in the live migration of instances with SR-IOV-based network interfaces.

In Red Hat OpenStack Platform 16.0, you are now able to push, list, delete, and show (show metadata) images on the local registry.

In Red Hat OpenStack Platform 16.0, a new Orchestration service (heat) deployment parameter is available that enables administrators to turn on the nova metadata service on cell controllers:

parameter_defaults:
   NovaLocalMetadataPerCell: True

This new parameter automatically directs traffic from the OVN metadata agent on the cell computes to the nova metadata API service hosted on the cell controllers.

Depending on the RHOSP topology, the ability to run the metadata service on cell controllers can reduce the traffic on the central control plane.

With this enhancement, you can schedule dedicated (pinned) and shared (unpinned) instances on the same Compute node using the following parameters:

Defining multiple route tables and assigning routes to particular tables is a technology preview in Red Hat OpenStack Platform 16.0.

Policy-based routing uses route tables where, on a host with multiple links, you can send traffic through a particular interface depending on the source address.

You can also define route rules on a per-interface basis, as shown in this example:

network_config:
  -
    type: route_table
    name: custom
    table_id: 200
  -
    type: interface
    name: em1
    use_dhcp: false
    addresses:
    -
      ip_netmask: 192.0.2.1/24
    routes:
      -
        ip_netmask: 10.1.3.0/24
        next_hop: 192.0.2.5
        table: 200  # Use table ID or table name
    rules:
      - rule: "iif em1 table 200"
        comment: "Route incoming traffic to em1 with table 200"
      - rule: "from 192.0.2.0/24 table 200"
        comment: "Route all traffic from 192.0.2.0/24 with table 200"
      - rule: "add blackhole from 172.19.40.0/24 table 200"
      - rule: "add unreachable iif em1 from 192.168.1.0/24"

In Red Hat OpenStack Platform 16.0, a technology preview has been added for booting bare metal machines from virtual media using the Bare Metal Provisioning service (ironic).

If the baseboard management controller (BMC) for a machine supports Redfish hardware management protocol and virtual media service, ironic can instruct the BMC to pull a bootable image and “insert” it into a virtual drive on a node. The node can then boot from that virtual drive into the operating system residing on the image. Ironic hardware types based on the Redfish API support deploy, rescue (with a limitation), and boot (user) images over virtual media.

The major advantage of virtual media boot is that the insecure and unreliable TFTP image transfer phase of the PXE boot protocol suite is replaced by secure HTTP transport.

In Red Hat OpenStack Platform 16.0, a Workflow service (mistral) task is in technology preview that allows you to implement password rotation by doing the following:

Execute the rotate-password workflow to generate new passwords and store them in the plan environment.

Redeploy your overcloud.

You can also obtain your passwords after you have changed them.

To implement password rotation, follow these steps:

In Red Hat OpenStack Platform 16.0, a technology preview has been added to the Orchestration service (heat) for rsyslog changes:

In Red Hat OpenStack Platform 16.0, you can use director to include in-flight validations in the service template. This feature is a technology preview in RHOSP 16.0. Additions can be inserted at the end of the step to be checked, or at the beginning of the next step.

In this example, a validation is performed to ensure that the rabbitmq service is running after its deployment:

deploy_steps_tasks:
  # rabbitmq container is supposed to be started during step 1
  # so we want to ensure it's running during step 2
  - name: validate rabbitmq state
    when: step|int == 2
    tags:
      - opendev-validation
      - opendev-validation-rabbitmq
    wait_for_connection:
      host: {get_param: [ServiceNetMap, RabbitmqNetwork]}
      port: 5672
      delay: 10

Heat enables you to include existing validations from the openstack-tripleo-validations roles:

deploy_steps_tasks:
  - name: some validation
    when: step|int == 2
    tags:
      - opendev-validation
      - opendev-validation-rabbitmq
    include_role:
      role: rabbitmq-limits
    # We can pass vars to included role, in this example
    # we override the default min_fd_limit value:
    vars:
      min_fd_limit: 32768

You can find the definition of the rabbitmq-limits role here: https://opendev.org/openstack/tripleo-validations/src/branch/stable/train/roles/rabbitmq_limits/tasks/main.yml

Here is an example of using the existing service health check:

deploy_steps_tasks:
  # rabbitmq container is supposed to be started during step 1
  # so we want to ensure it's running during step 2
  - name: validate rabbitmq state
    when: step|int == 2
    tags:
      - opendev-validation
      - opendev-validation-rabbitmq
    command: >
      podman exec rabbitmq /openstack/healthcheck

In Red Hat OpenStack Platform 16.0, a technology preview has been added to the Orchestration service (heat). A new parameter, NovaSchedulerQueryImageType, has been added that controls the Compute service (nova) placement and scheduler components query placement for image type (scheduler/query_placement_for_image_type_support).

When set to true (the default), NovaSchedulerQueryImageType excludes compute nodes that do not support the disk format of the image used in a boot request.

For example, the libvirt driver uses Red Hat Ceph Storage as an ephemeral back end, and does not support qcow2 images (without an expensive conversion step). In this case, enabling NovaSchedulerQueryImageType ensures that the scheduler does not send requests to boot a qcow2 image to compute nodes that use Red Hat Ceph Storage.

You can now forward the traffic from a TCP, UDP, or other protocol port of a floating IP address to a TCP, UDP, or other protocol port associated to one of the fixed IP addresses of a neutron port. Forwarded traffic is managed by an extension to the neutron API and by an OpenStack Networking plug-in. A floating IP address can have more than one forwarding definition configured. However, you cannot forward traffic for IP addresses that have a pre-existing association to an OpenStack Networking port. Traffic can only be forwarded for floating IP addresses that are managed by centralized routers on the network (legacy, HA, and DVR+HA).

To forward traffic for a port of a floating IP address, use the following OpenStack Networking plug-in command:

openstack floating ip port forwarding create
--internal-ip-address <internal-ip-address>
--port <port>
--internal-protocol-port <port-number>
--external-protocol-port <port-number>
--protocol <protocol>
<floating-ip>

--internal-ip-address <internal-ip-address> The fixed, IPv4, internal IP address of the neutron port that will receive the forwarded traffic.

--port <port> The name or ID of the neutron port that will receive the forwarded traffic.

--internal-protocol-port <port-number> The protocol port number of the neutron, fixed IP address that will receive the forwarded traffic.

--external-protocol-port <port-number> The protocol port number of the port of the floating IP address that will forward its traffic.

--protocol <protocol> The protocol that the port of the floating IP address uses (for example, TCP, UDP).

<floating-ip> The floating IP (IP address or ID) of the port that will forward its traffic.

Here is an example:

openstack floating ip port forwarding create \
    --internal-ip-address 192.168.1.2 \
    --port f7a08fe4-e79e-4b67-bbb8-a5002455a493 \
    --internal-protocol-port 18343 \
    --external-protocol-port 8343 \
    --protocol tcp \
    10.0.0.100

Nova-compute ironic driver tries to update BM node while the node is being cleaned up. The cleaning takes approximately five minutes but nova-compute attempts to update the node for approximately two minutes. After timeout, nova-compute stops and puts nova instance into ERROR state.

As a workaround, set the following configuration option for nova-compute service:

[ironic]
api_max_retries = 180

As a result, nova-compute continues to attempt to update BM node longer and eventually succeeds.

There is a known issue where migrations from ML2/OVS to OVN fail. The failure is caused by the new protective mechanism in Red Hat OpenStack Platform director to prevent upgrades while changing mechanism drivers.

For the workaround, see "Preparing for the migration," in the "Networking with Open Virtual Network" guide.

Red Hat OpenStack Platform deployments that use the Linux bridge ML2 driver and agent are unprotected against Address Resolution Protocol (ARP) spoofing. The version of Ethernet bridge frame table administration (ebtables) that is part of Red Hat Enterprise Linux 8 is incompatible with the Linux bridge ML2 driver.

The Linux Bridge ML2 driver and agent were deprecated in Red Hat OpenStack Platform 11, and should not be used.

Red Hat recommends that you use instead the ML2 Open Virtual Network (OVN) driver and services, the default deployed by Red Hat OpenStack Platform director.

Replacement of an overcloud Controller might cause swift rings to become inconsistent across nodes. This can result in decreased availability of Object Storage service. If this happens, log in to the previously existing Controller node using SSH, deploy the updated rings, and restart the Object Storage containers:

(undercloud) [stack@undercloud-0 ~]$ source stackrc
(undercloud) [stack@undercloud-0 ~]$ nova list
...
| 3fab687e-99c2-4e66-805f-3106fb41d868 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.17 |
| a87276ea-8682-4f27-9426-6b272955b486 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.38 |
| a000b156-9adc-4d37-8169-c1af7800788b | controller-3 | ACTIVE | -          | Running     | ctlplane=192.168.24.35 |
...

(undercloud) [stack@undercloud-0 ~]$ for ip in 192.168.24.17 192.168.24.38 192.168.24.35; do ssh $ip 'sudo podman restart swift_copy_rings ; sudo podman restart $(sudo podman ps -a --format="{{.Names}}" --filter="name=swift_*")'; done

There is a known issue where Red Hat OpenStack Platform 16.0, where metadata information required for configuring OpenStack instances is not available, and they might be started without connectivity.

An ordering issue causes the haproxy wrappers not to be updated for the ovn_metadata_agent.

A possible workaround is for the cloud operator to run the following Ansible command to restart the ovn_metadata_agent on select nodes after the update, to ensure that the ovn_metadata_agent is using an updated version of the haproxy wrapper script:

`ansible -b <nodes> -i /usr/bin/tripleo-ansible-inventory -m shell -a "status=`sudo systemctl is-active tripleo_ovn_metadata_agent`; if test \"$status\" == \"active\"; then sudo systemctl restart tripleo_ovn_metadata_agent; echo restarted; fi"`

In the earlier Ansible command, nodes may be a single node (for example, compute-0), all computes (for example, compute*) or "all".

As the ovn_metadata_agent is most commonly found on compute nodes, the following Ansible command restarts the agent for all compute nodes in the cloud:

`ansible -b compute* -i /usr/bin/tripleo-ansible-inventory -m shell -a "status=`sudo systemctl is-active tripleo_ovn_metadata_agent`; if test \"$status\" == \"active\"; then sudo systemctl restart tripleo_ovn_metadata_agent; echo restarted; fi"`

After you restart the ovn_metadata_agent services, they use the updated haproxy wrapper script, which enables them to provide metadata to VMs when they are started. Affected VMs already running should behave normally when they are restarted after the workaround has been applied.

There is a known issue in Red Hat OpenStack 16.0, where KVM guests do not start on IBM POWER8 systems unless the simultaneous multithreading (SMT) control is disabled. SMT is not disabled automatically.

The workaround is to execute sudo ppc64_cpu --smt=off on any IBM POWER8 compute nodes after deploying the overcloud, and any subsequent reboots.

In Red Hat OpenStack 16.0, there is a known issue where the command, "openstack network agent list," intermittently indicates that the OVN agents are down, when the agents are actually alive and the cloud is operational.

The affected agents are: OVN Controller agent, OVN Metadata agent, and OVN Controller Gateway agent.

There is currently no workaround for this issue. You should ignore the output of the "openstack network agent list" command.

There is a known issue for OpenStack Networking (neutron) where all instances created inherit the dns_domain value associated with the network, and not the dns_domain value configured for an internal DNS.

The cause of this issue is that the network dns_domain attribute overrides the neutron dns_domain config option.

To avoid this issue, do not set the dns_domain attribute for the network, if you want to use the internal DNS feature.

To allow Placement services deployed on the Controller node to be accessed by the neutron_api service, as required when using the Novacontrol role, add the following hieradata configuration to your Controller environment file:

service_config_settings:
     placement:
         neutron::server::placement::password: <Nova password>
  neutron::server::placement::www_authenticate_uri: <Keystone Internal API URL>
  neutron::server::placement::project_domain_name: 'Default'
  neutron::server::placement::project_name: 'service'
  neutron::server::placement::user_domain_name: 'Default'
  neutron::server::placement::username: nova
  neutron::server::placement::auth_url: <Keystone Internal API URL>
  neutron::server::placement::auth_type: 'password'
  neutron::server::placement::region_name: <Keystone Region>

For more information on using Puppet to customizing hieradata for roles, see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/advanced_overcloud_customization/index#sect-Customizing_Puppet_Configuration_Data.

There is a known issue for the Red Hat OpenStack Platform Load-balancing service: the containers octavia_api and octavia_driver_agent fail to start when rebooting a node.

The cause for this issue is that the directory, /var/run/octavia, does not exist when the node is rebooted.

To fix this issue, add the following line to the file, /etc/tmpfiles.d/var-run-octavia.conf:

d /var/run/octavia 0755 root root - -

In Red Hat OpenStack Platform 16.0, there is a known issue when ansible-playbook can sometimes fail during configuration of the overcloud nodes. The cause for the failure is the tripleo-admin user is not authorized for ssh. Furthermore, an openstack overcloud deploy command argument, --stack-only, no longer runs the enable ssh admin workflow to authorize the tripleo-admin user.

The workaround is to use the openstack overcloud admin authorize command to run the enable ssh admin workflow on its own when using --stack-only and the manual config-download commands. for more information, see "Separating the provisioning and configuration processes" in the Director Installation and Usage guide.

There is a known issue in Red Hat OpenStack Platform 16.0, when nodes experiencing hard (ungraceful) shutdowns put containers—​that were previously running---in a "Created" state in podman when the node is turned back on.

The reason for this issue is that the metadata agent fails to spawn a new container because it already exists in the "Created" state. The haproxy side-car container wrapper script expects containers to be in only the "Exited" state, and does not cleanup containers in the "Created" state.

The possible workaround is for the cloud operator to run the following Ansible ad-hoc command to clean up all haproxy containers in the "Created" state. You must run this Ansible ad-hoc command from the undercloud on particular node, on a group of nodes, or on the whole cluster:

`ansible -b <nodes> -i /usr/bin/tripleo-ansible-inventory -m shell -a "podman ps -a --format {{'{{'}}.ID{{'}}'}} -f name=haproxy,status=created | xargs podman rm -f || :"`

In the earlier Ansible ad-hoc command, nodes can be a single host from the inventory, a group of hosts, or "all".

Here is an example of running the command on compute-0:

`ansible -b compute-0 -i /usr/bin/tripleo-ansible-inventory -m shell -a "podman ps -a --format {{'{{'}}.ID{{'}}'}} -f name=haproxy,status=created | xargs podman rm -f || :"`

After running the Ansible ad-hoc command, the metadata-agent should then spawn a new container for the given network.

In Red Hat OpenStack Platform 16.0, the controller-v6.yaml file is removed. The routes that were defined in controller-v6.yaml are now defined in controller.yaml. (The controller.yaml file is a NIC configuration file that is rendered from values set in roles_data.yaml.)

Previous versions of Red Hat OpenStack director, included two routes: one for IPv6 on the External network (default) and one for IPv4 on the Control Plane.

To use both default routes, make sure that the controller definition in roles_data.yaml contains both networks in default_route_networks (for example, default_route_networks: ['External', 'ControlPlane']).

In Red Hat OpenStack Platform 16.0, the following environment file has been removed: /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-bootstrap-environment-rhel.yaml.

This environment file was previously used when using pre-provisioned nodes. It was deprecated in a previous RHOSP release, and now it has been removed.

To allow the neutron_api service to access Placement services on Controller nodes, for example, when you use the Novacontrol role, add the following hieradata configuration to your Controller environment file:

service_config_settings:
     placement:
         neutron::server::placement::password: <Nova password>
  neutron::server::placement::www_authenticate_uri: <Keystone Internal API URL>
  neutron::server::placement::project_domain_name: 'Default'
  neutron::server::placement::project_name: 'service'
  neutron::server::placement::user_domain_name: 'Default'
  neutron::server::placement::username: nova
  neutron::server::placement::auth_url: <Keystone Internal API URL>
  neutron::server::placement::auth_type: 'password'
  neutron::server::placement::region_name: <Keystone Region>

This feature enables the Red Hat OpenStack Platform director to deploy the Shared File System (manila) with an external Ceph Storage cluster. In this type of deployment, Ganesha still runs on the Controller nodes that Pacemaker manages using an active-passive configuration.

This feature is supported with Ceph Storage 4.1 or later.

The Service Telemetry Framework (STF) release v1.0 is now available for general availability. STF provides the core components for a monitoring application framework for Red Hat OpenStack Platform (RHOSP). It is a data storage component deployed as an application on top of OpenShift 4.x and is managed by the Operator Lifecycle Manager. Data transport for metrics and events is provided using AMQ Interconnect.

The release of STF v1.0 replaces and deprecates the Technology Preview version.

With this update, you can now enable Red Hat Ceph Storage Dashboard with the Red Hat OpenStack Platform director. The Red Hat Storage Ceph Dashboard is a built-in, web-based Ceph management and monitoring application to visualise and monitor various aspects in your cluster.

Ceph Dashboard requires Red Hat Ceph Storage 4.1 or later.

RHOSP 16.0 works only with RHEL 8.1. Ensure that all the hosts of your OSP deployment are pinned to RHEL 8.1 before running the update.

See "Locking the environment to a Red Hat Enterprise Linux release" [1] in the guide "Keeping Red Hat OpenStack Platform Updated."

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/keeping_red_hat_openstack_platform_updated/preparing-for-a-minor-update#locking-the-environment-to-a-red-hat-enterprise-linux-release

There is a known issue for the Red Hat OpenStack Platform Load-balancing service: the containers octavia_api and octavia_driver_agent fail to start when rebooting a node.

The cause for this issue is that the directory, /var/run/octavia, does not exist when the node is rebooted.

To fix this issue, add the following line to the file, /etc/tmpfiles.d/var-run-octavia.conf:

d /var/run/octavia 0755 root root - -

A Grafana Ceph 4.1 dependency causes Ceph dashboard bugs. The Ceph dashboard requires Ceph 4.1 and a Grafana container based on ceph4-rhel8. Presently, Red Hat supports ceph3-rhel7.3. This discrepancy causes the following dashboard bugs:

Because of a core OVN bug, virtual machines with floating IP (FIP) addresses cannot route to other networks in an ML2/OVN deployment with distributed virtual routing (DVR) enabled. Core OVN sets a bad next hop when routing SNAT IPv4 traffic from a VM with a floating ip with DVR enabled. Instead of the gateway IP, OVN sets the destination IP. As a result, the router sends an ARP request for an unknown IP instead of routing it to the gateway.

Before deploying a new overcloud with ML2/OVN, disable DVR by setting NeutronEnableDVR: false in an environment file.

If you have ML2/OVN in an existing deployment, perform the following steps: