Chapter 3. Release information

Before this update, baremetal node introspection failed with an error and did not retry, when the node had a transient lock on it.

With this update, you can perform introspection even when the node has a lock.

Before this update in Red Hat OpenStack Platform (RHOSP) 17.0 Beta, Networking service (neutron) requests could fail with a 504 Gateway Time-out if they occurred when the Networking service reconnected to ovsdb-server. These reconnections could happen during failovers or through ovsdb-server leader transfers during database compaction.

If neutron debugging was enabled, the Networking service rapidly logged a large number of OVSDB transaction returned TRY_AGAIN" DEBUG messages, until the transaction timed out with an exception.

With this update, the reconnection behavior is fixed to handle this condition, with a single retry of the transaction until a successful reconnection.

In Red Hat OpenStack Platform (RHOSP) 17.0 GA, non-admin users have access to new parameters when they run the openstack server list command:

With this update, the default machine type is RHEL9.0-based Q35 pc-q35-rhel9.0.0, with the following enhancements:

In Red Hat OpenStack Platform (RHOSP) 17.0 GA, the openstack tripleo validator show command has a new parameter, --limit <number>, that enables you to limit the number of validations that TripleO displays. The default value is to display the last 15 validations.

For more information, see tripleo validator show history.

With this update, the Validation Framework provides a configuration file in which you can set parameters for particular use. You can find an example of this file at the root of the code source or in the default location: /etc/validation.cfg.

You can use the default file in /etc/ or use your own file and provide it to the CLI with the argument --config.

When you use a configuration file there is an order for the variables precedence. The following order is the order of variable precedence:

In Red Hat OpenStack Platform 17.0, a technology preview is available to configure counting of quota usage of cores and ram by querying placement for resource usage and instances from instance mappings in the API database, instead of counting resources from separate cell databases. This makes quota usage counting resilient to temporary cell outages or poor cell performance in a multi-cell environment.

Set the following configuration option to count quota usage from placement:

parameter_defaults:
  ControllerExtraConfig:
    nova::config::nova_config:
      quota/count_usage_from_placement:
        value: 'True'

In Red Hat OpenStack Platform 17.0, a technology preview is available for cold migrating and resizing instances that have vGPUs.

For a known issue affecting the vGPU Technology Preview, see https://bugzilla.redhat.com/show_bug.cgi?id=2116979.

In an ML2/OVS deployment, Open vSwitch (OVS) does not support offloading OpenFlow rules that have the skb_priority, skb_mark, or output queue fields set. Those fields are needed to provide quality-of-service (QoS) support for virtio ports.

If you set a minimum bandwidth rule for a virtio port, the Neutron Open vSwitch agent marks the traffic of this port with a Packet Mark Field. As a result, this traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means no traffic can be offloaded.

As a workaround, if your environment includes OVS hardware offload ports, disable the packet marking in the nodes that require hardware offloading. After you disable the packet marking, it will not be possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules will still be available.

In the configuration file, set the disable_packet_marking flag to true. After you edit the configuration file, you must restart the neutron_ovs_agent container. For example:

$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
  [ovs]
  disable_packet_marking=True

There is currently a known issue wherein long-running operations can cause the ovsdb connection to time out causing reconnects. These time outs can then cause the nova-compute agent to become unresponsive. Workaround: You can use the command-line client instead of the default native python bindings. Use the following parameters in your heat templates to use the command-line client:

parameter_defaults:
  ComputeExtraConfig:
    nova:os_vif_ovs:ovsdb_interface => 'vsctl'

A known issue in RHOSP 17.0 prevents the default mechanism for selecting the hypervisor fully qualified domain name (FQDN) from being set properly if the resource_provider_hypervisors heat parameter is not set. This causes the SRIOV or OVS agent to fail to start.

Workaround: Specify the hypervisor FQDN explicitly in the heat template. The following is an example of setting this parameter for the SRIOV agent:

ExtraConfig: neutron::agents::ml2::sriov::resource_provider_hypervisors: "enp7s0f3:%{hiera('fqdn_canonical')},enp5s0f0:%{hiera('fqdn_canonical')}".

There is currently a known issue that causes tuned kernel configurations to not be applied after initial provisioning.

Workaround: You can use the following custom playbook to ensure that the tuned kernel command line arguments are applied. Save the following playbook as /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-reset-blscfg.yaml on the undercloud node:

- name: Reset BLSCFG of compute node(s) meant for NFV deployments
  hosts: allovercloud
  any_errors_fatal: true
  gather_facts: true

  pre_tasks:
    - name: Wait for provisioned nodes to boot
      wait_for_connection:
        timeout: 600
        delay: 10

  tasks:
    - name: Reset BLSCFG flag in grub file, if it is enabled
      become: true
      lineinfile:
        path: /etc/default/grub
        line: "GRUB_ENABLE_BLSCFG=false"
        regexp: "^GRUB_ENABLE_BLSCFG=.*"
        insertafter: '^GRUB_DISABLE_RECOVERY.*'

Configure the role in the node definition file, overcloud-baremetal-deploy.yaml, to run the cli-overcloud-node-reset-blscfg.yaml playbook before the playbook that sets the kernelargs:

- name: ComputeOvsDpdkSriov
  count: 2
  hostname_format: computeovsdpdksriov-%index%
  defaults:
    networks:
    - network: internal_api
      subnet: internal_api_subnet
    - network: tenant
      subnet: tenant_subnet
    - network: storage
      subnet: storage_subnet
    network_config:
      template: /home/stack/osp17_ref/nic-configs/computeovsdpdksriov.j2
    config_drive:
      cloud_config:
        ssh_pwauth: true
        disable_root: false
        chpasswd:
          list: |-
            root:12345678
        expire: False
  ansible_playbooks:
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-reset-blscfg.yaml
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
      extra_vars:
        reboot_wait_timeout: 600
        kernel_args: 'default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-11,13-23'
        tuned_profile: 'cpu-partitioning'
        tuned_isolated_cores: '1-11,13-23'
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
      extra_vars:
        memory_channels: '4'
        lcore: '0,12'
        pmd: '1,13,2,14,3,15'
        socket_mem: '4096'
        disable_emc: false
        enable_tso: false
        revalidator: ''
        handler: ''
        pmd_auto_lb: false
        pmd_load_threshold: ''
        pmd_improvement_threshold: ''
        pmd_rebal_interval: ''
        nova_postcopy: true

There is currently a known issue where the Swift API does not work and returns a 401 error when multiple Controller nodes are deployed and Ceph is enabled.

A workaround is available at https://access.redhat.com/solutions/6970061.

Live migration fails when executing the QEMU command migrate-set-capabilities. This is because the post-copy feature that is enabled by default is not supported.

Choose one of the following workaround options:

Sensubility does not have permission to access /run/podman/podman.sock, which causes the container health check to fail to send the service container status data to Service Telemetry Framework (STF).

Workaround: Run the following command on all overcloud nodes after deployment: sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman

Result: User collectd gets access to /run/podman path recursively allowing sensubility to connect to podman.

In Red Hat OpenStack Platform (RHOSP) 17.0 GA, there is a known issue where ML2/OVN deployments fail to automatically create DNS records with the RHOSP DNS service (designate). The cause for this problem is that the required Networking service (neutron) extension, dns_domain_ports, is not present.

Workaround: currently there is no workaround, but the fix has been targeted for a future RHOSP release.

In Red Hat OpenStack Platform (RHOSP) 17.0, the DNS service (designate) and the Load-balancing service (octavia) are misconfigured for high availability. The RHOSP Orchestration service (heat) templates for these services use the non-Pacemaker version of the Redis template.

Workaround: include environments/ha-redis.yaml in the overcloud deploy command after the enable-designate.yaml and octavia.yaml environment files.

In Red Hat OpenStack Platform (RHOSP) 17.0 GA, there is a known issue where the Free Range Router (FRR) container does not start after the host on which it resides is rebooted. This issue is caused by a missing file in the BGP configuration.

Workaround: create the file, /etc/tmpfiles.d/run-frr.conf, and add the following line:

d /run/frr 0750 root root - -

After you make this change, tmpfiles recreates /run/frr after each reboot and the FRR container can start.

The Derived Parameters feature is deprecated. It will be removed in a future release. The Derived Parameters feature is configured using the --plan-environment-file option of the openstack overcloud deploy command.

Workaround / Migration Instructions

HCI overclouds require system tuning. There are many different options for system tuning. The Derived Parameters functionality tuned systems with director by using hardware inspection data and set tuning parameters using the --plan-environment-file option of the openstack overcloud deploy command. The Derived Parameters functionality is deprecated in Release 17.0 and is removed in 17.1.

The following parameters were tuned by this functionality:

The ML2/OVS mechanism driver is deprecated in RHOSP 17.0.

Over several releases, Red Hat is replacing ML2/OVS with ML2/OVN. For instance, starting with RHOSP 15, ML2/OVN became the default mechanism driver.

Support is available for the deprecated ML2/OVS mechanism driver through the RHOSP 17 releases. During this time, the ML2/OVS driver remains in maintenance mode, receiving bug fixes and normal support, and most new feature development happens in the ML2/OVN mechanism driver.

In RHOSP 18.0, Red Hat plans to completely remove the ML2/OVS mechanism driver and stop supporting it.

If your existing Red Hat OpenStack Platform (RHOSP) deployment uses the ML2/OVS mechanism driver, start now to evaluate a plan to migrate to the mechanism driver. Migration is supported in RHOSP 16.2 and will be supported in RHOSP 17.1. Migration tools are available in RHOSP 17.0 for test purposes only.

Red Hat requires that you file a proactive support case before attempting a migration from ML2/OVS to ML2/OVN. Red Hat does not support migrations without the proactive support case. See How to submit a Proactive Case.

Technology preview support was added in RHOSP 16.1 for configuring NVDIMM Compute nodes to provide persistent memory for instances. Red Hat has removed support for persistent memory from RHOSP 17.0 and future releases in response to the announcement by the Intel Corporation on July 28, 2022 that they are discontinuing investment in their Intel® Optane™ business:

Cloud operators must ensure that no instances use the vPMEM feature before upgrading to 17.1.

Before this update, in Red Hat OpenStack Platform (RHOSP) 17.0, Networking service (neutron) requests sometimes failed with a 504 Gateway Time-out if the request was made when the Networking service reconnected to ovsdb-server. These reconnections sometimes happened during failovers or through ovsdb-server leader transfers during database compaction.

If neutron debugging was enabled, the Networking service rapidly logged a large number of OVSDB transaction-returned "TRY_AGAIN" DEBUG messages, until the transaction timed out with an exception.

With this update, the reconnection behavior is fixed to handle this condition, with a single retry of the transaction until a successful reconnection.

A known issue in RHOSP 17.0 prevents the default mechanism for selecting the hypervisor fully qualified domain name (FQDN) from being set properly if the resource_provider_hypervisors heat parameter is not set. This causes the single root I/O virtualization (SR-IOV) or Open vSwitch (OVS) agent to fail to start.

Workaround: Specify the hypervisor FQDN explicitly in the heat template. The following is an example of setting this parameter for the SRIOV agent:

ExtraConfig: neutron::agents::ml2::sriov::resource_provider_hypervisors: "enp7s0f3:%{hiera('fqdn_canonical')},enp5s0f0:%{hiera('fqdn_canonical')}".

There is currently a known issue where the ovn/ovsdb_probe_interval value is not configured in the file ml2_conf.ini with the value specified by OVNOvsdbProbeInterval because a patch required to configure the neutron server based on OVNOvsdbProbeInterval is not included in 17.0.1.

Workaround: Deployments that use OVNOvsdbProbeInterval must use ExtraConfig hooks in the following manner to configure the neutron server:

parameter_defaults:
  OVNOvsdbProbeInterval: <probe interval in milliseconds>
  ControllerExtraConfig:
    neutron::config::plugin_ml2_config:
      ovn/ovsdb_probe_interval:
        value: <probe interval in milliseconds>

There is currently a known issue that causes tuned kernel configurations to not be applied after initial provisioning.

Workaround: You can use the following custom playbook to ensure that the tuned kernel command line arguments are applied. Save the following playbook as /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-reset-blscfg.yaml on the undercloud node:

- name: Reset BLSCFG of compute node(s) meant for NFV deployments
  hosts: allovercloud
  any_errors_fatal: true
  gather_facts: true

  pre_tasks:
    - name: Wait for provisioned nodes to boot
      wait_for_connection:
        timeout: 600
        delay: 10

  tasks:
    - name: Reset BLSCFG flag in grub file, if it is enabled
      become: true
      lineinfile:
        path: /etc/default/grub
        line: "GRUB_ENABLE_BLSCFG=false"
        regexp: "^GRUB_ENABLE_BLSCFG=.*"
        insertafter: '^GRUB_DISABLE_RECOVERY.*'

Configure the role in the node definition file, overcloud-baremetal-deploy.yaml, to run the cli-overcloud-node-reset-blscfg.yaml playbook before the playbook that sets the kernelargs:

- name: ComputeOvsDpdkSriov
  count: 2
  hostname_format: computeovsdpdksriov-%index%
  defaults:
    networks:
    - network: internal_api
      subnet: internal_api_subnet
    - network: tenant
      subnet: tenant_subnet
    - network: storage
      subnet: storage_subnet
    network_config:
      template: /home/stack/osp17_ref/nic-configs/computeovsdpdksriov.j2
    config_drive:
      cloud_config:
        ssh_pwauth: true
        disable_root: false
        chpasswd:
          list: |-
            root:12345678
        expire: False
  ansible_playbooks:
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-reset-blscfg.yaml
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
      extra_vars:
        reboot_wait_timeout: 600
        kernel_args: 'default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-11,13-23'
        tuned_profile: 'cpu-partitioning'
        tuned_isolated_cores: '1-11,13-23'
    - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
      extra_vars:
        memory_channels: '4'
        lcore: '0,12'
        pmd: '1,13,2,14,3,15'
        socket_mem: '4096'
        disable_emc: false
        enable_tso: false
        revalidator: ''
        handler: ''
        pmd_auto_lb: false
        pmd_load_threshold: ''
        pmd_improvement_threshold: ''
        pmd_rebal_interval: ''
        nova_postcopy: true

There is currently a known issue in RHOSP 17.0 where the Free Range Router (FRR) container does not start after the host on which it resides is rebooted. This issue is caused by a missing file in the BGP configuration. Workaround: Create the file, /etc/tmpfiles.d/run-frr.conf, and add the following line:

d /run/frr 0750 root root - -

After you make this change, tmpfiles recreates /run/frr after each reboot and the FRR container can start.