6.5. RHBA-2015:2651 — Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory

A missing dnsmasq lease file caused "instack-virt-setup" to fail when retrieving the host IP. This fix retrieves the IP address from a different source and "instack-virt-setup" now succeeds.

A missing dnsmasq lease file caused "instack-virt-setup" to fail when retrieving the host IP. This fix retrieves the IP address from a different source and "instack-virt-setup" now succeeds.

When updating an Overcloud, the director parsed each breakpoint individually. This added to the amount of time it took to update the Overcloud. This fix parses the whole stack once for all breakpoints, reducing the amount of time to update.

When updating an Overcloud, the director parsed each breakpoint individually. This added to the amount of time it took to update the Overcloud. This fix parses the whole stack once for all breakpoints, reducing the amount of time to update.

The director updates Heat parameters even if a stack-update operation failed. The Heat stack *Count parameters (e.g. ComputeCount) might not reflect the real number of nodes in an Overcloud. This caused the "overcloud node delete" command to delete an incorrect number of nodes. This fix modifies the "overcloud node delete" command to compute the current node count from the real number of servers in ResourceGroup instead of using stack parameters. Now the director deletes the correct number of nodes.

The director updates Heat parameters even if a stack-update operation failed. The Heat stack *Count parameters (e.g. ComputeCount) might not reflect the real number of nodes in an Overcloud. This caused the "overcloud node delete" command to delete an incorrect number of nodes. This fix modifies the "overcloud node delete" command to compute the current node count from the real number of servers in ResourceGroup instead of using stack parameters. Now the director deletes the correct number of nodes.

The timeout for Overcloud updates was set to 90 minutes. This caused incomplete updates that took longer than 90 minutes. This fix extends the timeout to 240 minutes. Longer updates now complete successfully.

The timeout for Overcloud updates was set to 90 minutes. This caused incomplete updates that took longer than 90 minutes. This fix extends the timeout to 240 minutes. Longer updates now complete successfully.

Heat templates lacked the *RemovalPolicies parameters. This meant it was not possible to delete specific nodes when using Heat templates  directly (i.e. not through Tuskar). This update adds the *RemovalPolicies parameters. Now a user can specify particular nodes to remove by setting the *RemovalPolicies parameters.

Heat templates lacked the *RemovalPolicies parameters. This meant it was not possible to delete specific nodes when using Heat templates  directly (i.e. not through Tuskar). This update adds the *RemovalPolicies parameters. Now a user can specify particular nodes to remove by setting the *RemovalPolicies parameters.

The os-apply-config tool included unit tests as part of its installation. These tests were meant to be omitted. This fix updates the spec file to remove these tests. Unit test are no longer included with os-apply-config.

The os-apply-config tool included unit tests as part of its installation. These tests were meant to be omitted. This fix updates the spec file to remove these tests. Unit test are no longer included with os-apply-config.

When registering new nodes and using the existing JSON file to store their info, the director would overwrite the capabilities of existing nodes. This fix skips the existing nodes. Now users can add new nodes to the original JSON file without erasing capabilities for existing nodes.

When registering new nodes and using the existing JSON file to store their info, the director would overwrite the capabilities of existing nodes. This fix skips the existing nodes. Now users can add new nodes to the original JSON file without erasing capabilities for existing nodes.

This enhancement adds support for Fujitsu's iRMC Ironic driver in the director. The director now controls the power management of iRMC nodes in the Overcloud.

This enhancement adds support for Fujitsu's iRMC Ironic driver in the director. The director now controls the power management of iRMC nodes in the Overcloud.

The base RPMs included unit tests. This meant the director had  unnecessary files that consumed extra space. This fix removes the tests from the base RPMs. Unit tests are now only shipped with the source RPMs.

The base RPMs included unit tests. This meant the director had  unnecessary files that consumed extra space. This fix removes the tests from the base RPMs. Unit tests are now only shipped with the source RPMs.

The "os-collect-config" service on the Overcloud restarted on an RPM update. This caused Overcloud updates to fail. This fix changes the behavior so that "os-collect-config" does not restart on an RPM update. The Overcloud updates now succeed after an update of "os-collect-config". Note that "os-collect-config" gracefully restarts itself when "os-refresh-config" runs, so the restart on update is not required.

The "os-collect-config" service on the Overcloud restarted on an RPM update. This caused Overcloud updates to fail. This fix changes the behavior so that "os-collect-config" does not restart on an RPM update. The Overcloud updates now succeed after an update of "os-collect-config". Note that "os-collect-config" gracefully restarts itself when "os-refresh-config" runs, so the restart on update is not required.

This enhancement adds Linux bonding configuration through the director. The director used only OVS and VLAN bonding previously. Linux bonding provides increased performance and additional bonding modes.

This enhancement adds Linux bonding configuration through the director. The director used only OVS and VLAN bonding previously. Linux bonding provides increased performance and additional bonding modes.

No regular database maintenance process existed in previous versions of the director. As a result, the director's database grew without limit. This fix adds a cronjob to flush expired tokens from the database. This cleans the database periodically and reduces its size.

No regular database maintenance process existed in previous versions of the director. As a result, the director's database grew without limit. This fix adds a cronjob to flush expired tokens from the database. This cleans the database periodically and reduces its size.

An issue with Red Hat Enterprise Linux 7 boot settings caused unpredictable NIC naming on Overcloud images. This fix updates the Overcloud images to use Red Hat Enterprise Linux 7.2, which contains a fix for this issue.

An issue with Red Hat Enterprise Linux 7 boot settings caused unpredictable NIC naming on Overcloud images. This fix updates the Overcloud images to use Red Hat Enterprise Linux 7.2, which contains a fix for this issue.

In rare cases, RabbitMQ fails to start on deployment. As a workaround, manually start RabbitMQ on nodes:

[stack@director ~]$ ssh heat-admin@192.168.0.20
[heat-admin@overcloud-controller-0 ~]$ pcs resource debug-start rabbitmq

Then rerun the deployment command on the director. The deployment now succeeds.

In rare cases, RabbitMQ fails to start on deployment. As a workaround, manually start RabbitMQ on nodes:

[stack@director ~]$ ssh heat-admin@192.168.0.20
[heat-admin@overcloud-controller-0 ~]$ pcs resource debug-start rabbitmq

Then rerun the deployment command on the director. The deployment now succeeds.

This enhancement upgrades the Overcloud image content to Red Hat Enterprise Linux 7.2 content, including the latest version of Pacemaker. The previous Overcloud image used Red Hat Enterprise Linux 7.1 content.

This enhancement upgrades the Overcloud image content to Red Hat Enterprise Linux 7.2 content, including the latest version of Pacemaker. The previous Overcloud image used Red Hat Enterprise Linux 7.1 content.

Rerunning the director configuration script failed while restoring an Undercloud. This fix makes the configuration script re-runnable without requiring database clean-up. Users can now rerun the director configuration script without failure.

Rerunning the director configuration script failed while restoring an Undercloud. This fix makes the configuration script re-runnable without requiring database clean-up. Users can now rerun the director configuration script without failure.

This feature allows the reapplication of Puppet manifests on a deployed Overcloud. This ensures the overcloud has the desired configuration or can recover accidentally amended or deleted configuration files.

To have Puppet run again on the Overcloud nodes, omit the "--templates" option but include the following two environments files at the beginning of your deployment:

* /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml
* /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml

For example:

$ openstack overcloud deploy -e ~/templates/overcloud-resource-registry-puppet.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml [additional arguments from initial deployment]

This feature allows the reapplication of Puppet manifests on a deployed Overcloud. This ensures the overcloud has the desired configuration or can recover accidentally amended or deleted configuration files.

To have Puppet run again on the Overcloud nodes, omit the "--templates" option but include the following two environments files at the beginning of your deployment:

* /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml
* /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml

For example:

$ openstack overcloud deploy -e ~/templates/overcloud-resource-registry-puppet.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml [additional arguments from initial deployment]

Old puppet manifests were reapplied during the update process when they should not have been. This had the potential to take the cluster services down in the Overcloud. The agent on the overcloud nodes caused the reapplication of the old Puppet manifests because their state was saved in tmpfs mounted directory under /var/run/. This directory is lost on reboot. This update moves the directory from /var/run/heat-config/deployed to /var/lib/heat-config/deployed, which allows the deployed state to persist across reboots.

Old puppet manifests were reapplied during the update process when they should not have been. This had the potential to take the cluster services down in the Overcloud. The agent on the overcloud nodes caused the reapplication of the old Puppet manifests because their state was saved in tmpfs mounted directory under /var/run/. This directory is lost on reboot. This update moves the directory from /var/run/heat-config/deployed to /var/lib/heat-config/deployed, which allows the deployed state to persist across reboots.

Pacemaker prevented Puppet from restarting services during an Overcloud update. This caused Puppet runs to fail, which caused the Overcloud update to fail. This fix puts Pacemaker into maintenance mode for the duration of the Puppet run during the Overcloud update. Pacemaker now allows Puppet to restart services and now the Overcloud update does not fail anymore.

Pacemaker prevented Puppet from restarting services during an Overcloud update. This caused Puppet runs to fail, which caused the Overcloud update to fail. This fix puts Pacemaker into maintenance mode for the duration of the Puppet run during the Overcloud update. Pacemaker now allows Puppet to restart services and now the Overcloud update does not fail anymore.

Configuration records for heat-config were stored under /var/run, which were lost on node reboot. The cause for this was reported and fix (see BZ#1278181) but had an impact on the OSP director; Puppet reapplied old manifests during an Overcloud update, which caused various Pacemaker cluster errors. The fix for BZ#1278181 moves the configuration records from /var/run/heat-config to /var/lib/heat-config. This fix also includes the "heat-config-rebuild-deployed" script to rebuild the configuration records. Now updates are possible without reapplying old Puppet manifests. Make sure to follow the documented update procedure, including running the "heat-config-rebuild-deployed" script on each node.

Configuration records for heat-config were stored under /var/run, which were lost on node reboot. The cause for this was reported and fix (see BZ#1278181) but had an impact on the OSP director; Puppet reapplied old manifests during an Overcloud update, which caused various Pacemaker cluster errors. The fix for BZ#1278181 moves the configuration records from /var/run/heat-config to /var/lib/heat-config. This fix also includes the "heat-config-rebuild-deployed" script to rebuild the configuration records. Now updates are possible without reapplying old Puppet manifests. Make sure to follow the documented update procedure, including running the "heat-config-rebuild-deployed" script on each node.

A missing SELinux rule caused Glance to return an "Invalid OpenStack Identity credentials" error. This fix adds the SELinux rule. Now Glance authenticates successfully.

A missing SELinux rule caused Glance to return an "Invalid OpenStack Identity credentials" error. This fix adds the SELinux rule. Now Glance authenticates successfully.

Due to various issues with updating the Overcloud heat stack, updates from 7.0 to 7.1 with network isolation would not work. This fix corrects these issues. Now you can update deployments from 7.0 to 7.2 directly. Updates from 7.0 to 7.1 should be skipped. Ensure to to follow the documented update procedure, as additional Heat environment files are needed depending on the configuration of particular deployments.

Due to various issues with updating the Overcloud heat stack, updates from 7.0 to 7.1 with network isolation would not work. This fix corrects these issues. Now you can update deployments from 7.0 to 7.2 directly. Updates from 7.0 to 7.1 should be skipped. Ensure to to follow the documented update procedure, as additional Heat environment files are needed depending on the configuration of particular deployments.

Orphaned OpenStack Networking L3 agent keepalived processes were left running by OpenStack Networking's "netns-cleanup" script. As a result, the OpenStack Networking tenant router failover did not work during the Controller node update in the Overcloud. This fix ensures the keepalived processes are cleaned up properly during the Controller node update. As a result, OpenStack Networking tenant router failover works normally and the high availability of the tenant network is preserved.

Orphaned OpenStack Networking L3 agent keepalived processes were left running by OpenStack Networking's "netns-cleanup" script. As a result, the OpenStack Networking tenant router failover did not work during the Controller node update in the Overcloud. This fix ensures the keepalived processes are cleaned up properly during the Controller node update. As a result, OpenStack Networking tenant router failover works normally and the high availability of the tenant network is preserved.

Strict timeouts on starting and stopping services through Pacemaker caused updates to fail. This fix increases the timeouts for starting and stopping services through Pacemaker. Now services start and stop within the timeout limit and the Overcloud now succeeds.

Strict timeouts on starting and stopping services through Pacemaker caused updates to fail. This fix increases the timeouts for starting and stopping services through Pacemaker. Now services start and stop within the timeout limit and the Overcloud now succeeds.