Chapter 3. Release Information

This update adds a new user details page to the dashboard.  As a result you can click on the user ID in the Action Log to be taken directly to the user details page.

This update adds a new user details page to the dashboard.  As a result you can click on the user ID in the Action Log to be taken directly to the user details page.

The Time Series Database as a Service (gnocchi) and Aodh API endpoints now expose a `/healthcheck` HTTP endpoint on the REST API. Requesting this endpoint allows you to check the status of the service, and does not require authentication.

Automatic fencing setup can be used in director for easier High Availability deployments and upgrades. To benefit from the new feature, use the 'overcloud generate fencing' command.

The administrator needs to record the user credentials during the volume transfer operation and doing so by hand is inconvenient. 

With this update, a new button to download the credentials has been added to the volume transfer screen for saving the information easily. This allows the administrators to download and save a CSV file locally on their computer with the click of a button.

This enhancement adds the ability to automatically reschedule load balancers from LBaaS agents that the server detects as dead. Previously, load balancers could be scheduled and realized across multiple LBaaS agents, however if a hypervisor died, the load balancers scheduled
to that node would cease operation. With this update, these load balancers will be automatically rescheduled to a different agent. This feature is disabled by default and managed using `allow_automatic_lbaas_agent_failover`.

This enahncement implements the 'ProcessMonitor' class in the 'HaproxyNSDriver' class (v2), This class utilizes the 'external_process' module to monitor and respawn HAProxy processes if and when needed. The LBaaS agent (v2) loads 'external_process' related options and take a configured action when HAProxy dies unexpectedly.

This update adds support for the version 5.1.0 MapR plugin.

A disk can be in a variety of states which may cause director to fail when attempting to make the disk a Ceph OSD. In previous releases, a user could run a first-boot script to erase the disk and set a GPT label required by Ceph. With this release, a new default setting in Ironic will erase the disks when a node is set to available and a change in puppet-ceph will give the disk a GPT label if there is no GPT label on the disk.

This update provides enhancements to the CephFS Native Driver in conjunction with the core OpenStack File Share Service (manila) infrastructure. The CephFS Native driver now supports read-only shares and includes improvements to access rule update recovery mode.

To avoid memory bloat issues in the nova-api workers, pagination logic has been added to the simple-tenant-usage API extension.

With this enhancement, you can now enable the creation of non-public shares in the Dashboard.
You can configure Dashboard to hide the checkbox that enables users to mark shares as public during the creation process. The default option will be to create shares as private without the box checked.

With this enhancement, `glance-manage db purge` can now remove rows that are less than one day old. This was added because operators may need to run this operation on a regular basis.
As a result, the value of the `age_in_days` option can be set to `0`.

This release features the necessary puppet modules for deploying CephFS. This allows you to deploy the OpenStack Shared File System service (openstack-manila) with a CephFS back-end through the director.

This enhancement makes Time Series Database as a Service (gnocchi) available in the undercloud. Gnocchi provides metrics backend for Telemetry in OpenStack, and is enabled by default with the `enable_telemetry` flag set to `true`. All telemetry services can be disabled by setting `enable_telemetry=false` in `undercloud.conf`.

This release adds pagination support to avoid resource-consuming usage requests on systems with a large number of instances. The v2.40 microversion of the nova API simple-tenant-usage endpoints use new optional query parameters 'limit' and 'marker' for pagination. The 'marker' option sets the starting point and the 'limit' option sets the number of records to be displayed after the starting point. If 'limit' is not set, nova will use the configurable 'max_limit' (1000 by default). Although older microversions will not accept these new query parameters, they will start to enforce the max_limit and results may be truncated as a result. Consider using the new microversion to avoid DoS-like usage requests and potentially truncated responses.

With this update, different domain names can be used for the public and the internal networks.  To set domain names per network, use the following heat template parameters:

* CloudName: The DNS name of this cloud, for example:
      'ci-overcloud.tripleo.org'
* CloudNameInternal: The DNS name of this cloud's internal API endpoint, for example:
      'ci-overcloud.internalapi.tripleo.org'
* CloudNameStorage: The DNS name of this cloud's storage endpoint, for example:
      'ci-overcloud.storage.tripleo.org'
* CloudNameStorageManagement: The DNS name of this cloud's storage management endpoint, for example:
      'ci-overcloud.storagemgmt.tripleo.org'
* CloudNameCtlplane: The DNS name of this cloud's control plane endpoint, for example:
      'ci-overcloud.management.tripleo.org'

With this update, the user interface is now partially internationalized and available in Japanese and Simplified Chinese. Note that only the interface itself is internationalized at this stage. The strings that come from other services such as the parameters, templates, and environments for the TripleO Heat Templates, the validations, and the notifications are not yet internationalized.

Recent enhancements to the director requires changes to network interface configuration templates. The NIC configuration templates now use a script that calls the 'os-net-config utility' to configure the network on the overcloud nodes. There are three major changes to the NIC config templates:

* The 'OsNetConfigImpl' resource changed from a 'OS::Heat::StructuredConfig' resource type to 'OS::Heat::SoftwareConfig'. In addition, the resource now stores the 'network_config' property as a blob of text and passes the blob to the 'run-os-net-config.sh' script using the 'str_replace' (string replace) function. For example:

----
resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: ../../scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:
----

* The {get_input: <input>} constructor defined a default external bridge and interface. Now there are two special string values that substitute for the external bridge and interface. These are 'bridge_name' and 'interface_name' respectively. Instead of using '{get_input: bridge_name}' or '{get_input: interface_name}', use 'bridge_name' or 'interface_name'. For example:

----
              - type: ovs_bridge
                name: {get_input: bridge_name}
----
becomes:
----
              - type: ovs_bridge
                name: bridge_name
----

* The 'network_config' no longer uses curly braces. Instead, the {get_param: <param>} construct moves to a sub-level underneath the value being defined. For example:
----
                dns_servers: {get_param: DnsServers}
----
becomes:
----
                dns_servers:
                  get_param: DnsServers
----

See more examples in the "Network Isolation" chapter of the Advanced Overcloud Customizations guide.

With this update, Wake-On-LAN and AMT drivers have been removed from Ironic as they do not have, and are not planned to have, a third-party CI. They are still available from the unsupported ironic driver collection, found in the ironic-staging-drivers repository. If your ironic installation is using any driver based on those, you must install ironic-staging-drivers and change the driver on the affected nodes according to following list:

agent_amt -> pxe_amt_agent
pxe_amt -> pxe_amt_iscsi
agent_wol -> pxe_wol_agent
pxe_wol -> pxe_wol_iscsi

Because SELinux policies concerning launching instances with DPDK enabled are incomplete, launching instances using DPDK with SELinux in enforcing mode will cause the launch to fail and AVC denials will appear in /var/log/audit/audit.log* concerning openvswitch and svirt.

As a workaround, set SELinux to permissive on each compute node where DPDK is utilized as documented in section 4.4.1.2 here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Security-Enhanced_Linux-Working_with_SELinux-Changing_SELinux_Modes.html#sect-Security-Enhanced_Linux-Enabling_and_Disabling_SELinux-Disabling_SELinux

This will allow DPDK-enabled virtual machines to launch. This is a workaround and is expected to be temporary while the issue is investigated further.

Problem in detail:
In OSP10 (OvS2.5), following are the issues:
1) tuned is configured with wrong set of CPUs. Expected configuration is NeutronDpdkCoreList + NovaVcpuPinSet, but it has been configured as HostCpusList. 
2) In post-config, the -l of DPDK_OPTIONS is set as 0 and NeutronDpdkdCoreList is configured as pmd-cpu-mask

What needs to be corrected after update, manually?
1) Add the list of cpus to be isolated, which is NeutronDpdkCoreList + NovaVcpuPinSet to the tuned conf file.

TUNED_CORES="<list of CPUs"
sed -i 's/^isolated_cores=.*/isolated_cores=$TUNED_CORES/' $tuned_conf_path
tuned-adm profile cpu-partitioning

2) lcore mask after the update will be set to 0. Get the cpu mask with get_mask code from the first-boot script [1].
LCORE_MASK="<mask value output of get_mask"
ovs-vsctl --no-wait set Open_vSwitch . other-config:dpdk-lcore-mask=$LCORE_MASK

For security reasons, the Overcloud only allows SSH key-based access by default. You can set a root password on the disk image for the overcloud using the virt-customize tool, which is found in the Red Hat Enterprise Linux Extras channel. After installing the tool and downloading the Overcloud images, use the following command to change the root password:

$ virt-customize -a overcloud-full.qcow2 --root-password password:<my_root_password>

Perform this operation prior to uploading the images into glance with the "openstack overcloud image upload" command.

Only NICs which have a live connection to a switch port will be counted in the numbered NIC abstractions (nic1, nic2, etc). As a workaround, the director includes a script that pings the first controller on all interfaces from each node. If a node has a link disconnected at deployment time, this is detected so that it can be corrected. Another possible workaround is to use a mapping file for each host, which has a mapping of NIC number to physical NIC. If one or more Overcloud nodes do not configure correctly due to a down link, this is now detected and the Overcloud can redeploy.

Discovery fails if multiple network interfaces on a node are connected to the Provisioning network. Only one interface can connect to the Provisioning network. This interface cannot be part of a bond.

Pacemaker continuously crashes when the fencing device name and the host name are the same. To avoid this problem, add the "fence-" prefix or the "-fence" suffix to the name of the fencing device. With the names configured like this, the cluster works without errors.

When you enter the `openstack stack delete` command while the nodes are still being deployed, it is possible that they remain in that being-deployed state and will not be deleted. Consequently, you will not be able to deploy any new nodes onto the space, blocked by the undeleted nodes. To avoid this situation, wait until all the nodes have been deployed completely, before entering the `openstack stack delete` command. Alternatively, you can manually delete the nodes with the `nova delete <node>` command.

The 'openstack overcloud stack update' command has a lengthy startup time. The command can take a couple of minutes for an indication the command is working. This is normal behavior.

To implement the security groups trunk feature with neutron-openvswitch-agent, openvswitch firewall driver is required. This driver currently contains a bug 1444368 where ingress traffic is wrongly matched if there are two ports with same MAC address on different network segment on the same compute node.

As a result, if a subport has the same MAC address as its parent port, ingress traffic won't be matched correctly for one of the ports.

A workaround to achieve correctly handled traffic is to disable port-security on the parent port and subports.

For example, to disable port security on port with UUID 12345, you need to remove security groups associated with the port:
 openstack port set --no-security-group --disable-port-security 12345

Note that no security groups rules will be applied to that port and traffic will not be filtered or protected against ip/mac/arp spoofing.

A race condition may occur between Puppet and MongoDB services. Consequently, scaling out nodes running the MongoDB database fails and the overcloud stack will not update. Running the same deployment command again makes the MongoDB nodes scale out successfully.

Currently, the Red Hat OpenStack Platform director 10 with SRIOV overcloud deployment fails when using the NIC IDs (for example, nic1, nic2, nic3 and so on) in the compute.yaml file.

As a workaround, you need to use NIC names (for example, ens1f0, ens1f1, ens2f0, and so on) instead of the NIC IDs to ensure the overcloud deployment completes successfully.

Under certain circumstances, such as when OVS is upgraded, or when the network service is restarted, the OVS bridges were torn down and rebuilt. Consequently, the existing network flows were interrupted when this happened, causing network traffic to stop forwarding until the flows were rebuilt. This can take some time in a complex deployment.

In order to avoid any possible downtime, the control plane networks should not be placed on an OVS bridge. The Control Plane (Provisioning), Internal API, and Storage Management networks should instead be dedicated interfaces or VLAN interfaces that are not on a bridge. For instance, one interface or bond could contain the control plane VLANs, while another interface or bond can be placed on an OVS bridge for tenant network data.

As long as the control plane interfaces are not on an OVS bridge, any network downtime will be limited to the Tenant data plane.

Processing parameters, environments and templates is slightly different on the CLI than it is in the UI.
Consequently, the passwords generated automatically by the UI cannot be changed from the templates. If you want to use custom passwords, you have to set them manually in the UI as a parameter in the overall deployment configuration or by editing the role card. 
Alternatively, you can create a plan without auto-generated passwords by entering the '$ openstack overcloud plan create <my_plan> --disable-password-generation' on the CLI.  You will have to provide the passwords explicitly by using templates or manually through the UI.

Running the 'openstack overcloud deploy' command replaces the default 'overcloud' plan. If a user creates an overcloud with the CLI, deletes it, then creates a new overcloud with the web UI, the web UI uses the 'overcloud' plan from the CLI deployment. This can cause the web UI to include unwanted parameters from the previous overcloud deployment. As a workaround:

1. Ensure the 'user-environment.yaml' environment file is disabled when deploying a new overcloud.
2. Upload a new version of the plan (from the '/usr/share/openstack-tripleo-heat-template').

The director requires nodes in a managed state before running introspection. Although newly registered nodes are set to 'manageable' in the director's web UI, no option currently exists in the web UI to switch the nodes back to 'manageable' in case users require introspection at a later date. As a workaround, use the 'openstack baremetal node manage' command to switch the nodes to 'manageable' state.

Invalid cache files may cause os-collect-config to report 'ValueError: No JSON object could be decoded' and the service will fail to start. The cache files located in '/var/lib/os-collect-config/' should be valid json files. If they are are of size 0 or contain invalid json, remove the invalid files from '/var/lib/os-collect-config', otherwise they may prevent os-collect-config from starting.

Customers who upgraded from Red Hat OpenStack Platform 9 to version 10 are advised to wait for the first asynchronous release before upgrading to Red Hat OpenStack Platform 11, which will fix this known issue. The first asynchronous release typically happens within a few days of the GA release.

In Highly Available IPv6 deployments, virtual IPs used for RabbitMQ may move between controller hosts during an upgrade. A bug in the creation of these IPv6 IPs causes them to be used as source addresses for RabbitMQ's connections. As a result, RabbitMQ will crash and may be unable to automatically recover its cluster.

To return to normal operation, restart RabbitMQ on the affected controller hosts, as well as any services which depend on RabbitMQ and do not automatically reconnect.

Customers who upgraded from Red Hat OpenStack Platform 9 to version 10 are advised to wait for the first asynchronous release before upgrading to Red Hat OpenStack Platform 11, which will fix this known issue. The first asynchronous release typically happens within a few days of the GA release.

A design flaw issue was found in the Red Hat OpenStack Platform director use of TripleO to enable libvirtd based live migration. TripleO did not have support for secure live migration and no additional steps were taken to lock-down the libvirtd deployment by director. Libvirtd is deployed by default (by director) listening on 0.0.0.0 (all interfaces) with no-authentication or encryption. Anyone able to make a TCP connection to any compute host IP address, including 127.0.0.1, other loopback interface addresses or in some cases possibly addresses that have been exposed beyond the management interface, could use this to open a virsh session to the libvirtd instance and gain control of virtual machine instances or possibly take over the host.

Note that without the presence of additional flaws, this should not be accessible from tenant or external networks.

Users who are upgrading to Red Hat OpenStack Platform 11 from Red Hat OpenStack Platform 10 should first apply the relevant update that resolves this issue.

Red Hat OpenStack Platform 11 already contains this update as of general availability and no subsequent update is required.

For more information about this flaw and the accompanying resolution, see https://access.redhat.com/solutions/3022771.

If using a standalone Keystone node on OpenStack Platform 10, 'openstack-gnocchi-statsd' does not start correctly. This is because 'gnocchi' and 'keystone' services activate on the same step, which causes a race condition. 'gnocchi' fails authentication but does not retry. This issue is addressed in BZ#1447422.

The failed 'openstack-gnocchi-statsd' service causes OpenStack Platform 11 pre-upgrade check to fail, which means upgrades from OpenStack Platform 10 to 11 also fail if using a standalone Keystone role. As a work around, restart 'openstack-gnocchi-statsd' service on the overcloud before starting any upgrade steps. This enables the service correctly and allows for a successful upgrade. This only applies to OpenStack Platform 10 with a standalone Keystone node and does not impact other upgrade scenarios.

When using Red Hat Ceph Storage as a back end for both Block Storage (cinder) volumes and backups, any attempt to perform an incremental backup will result in a full backup instead, without any warning.

OpenStack command-line clients that use `python-requests` can not currently validate certificates that have an IP address in the SAN field.

The image_path parameter is no longer used. This update removes it from the undercloud configuration file.

The VPNaaS feature is deprecated with Red Hat OpenStack Platform 11 and is expected to be removed with Red Hat OpenStack Platform 12.

Neutron's Linux Bridge ML2 driver and agent are being deprecated with Red Hat OpenStack Platform 11 and are expected to be removed with Red Hat OpenStack Platform 12. The Open vSwitch (OVS) plug-in is the one deployed by default by the OpenStack Platform director, and is recommended by Red Hat for general usage.

The Ceilometer API service is deprecated in Red Hat OpenStack Platform 11. It is replaced by the Gnocchi, Aodh, and Panko APIs respectively. Users should begin to move away from the Ceilometer API and use the service APIs instead. In Red Hat OpenStack Platform 11, the Ceilometer API is installed and configured by default. In future releases, this API will be disabled by default, with the option to enable it only if required.

As of this release, Glance API V1 is no longer supported or available.