Chapter 15. Troubleshooting Director Issues

Understanding historical director deployment commands and arguments can be useful for troubleshooting and support. You can view this information in /home/stack/.tripleo/history.

In most cases, Heat shows the failed overcloud stack after the overcloud creation fails:

$ source ~/stackrc
(undercloud) $ openstack stack list --nested --property status=FAILED
+-----------------------+------------+--------------------+----------------------+
| id                    | stack_name | stack_status       | creation_time        |
+-----------------------+------------+--------------------+----------------------+
| 7e88af95-535c-4a55... | overcloud  | CREATE_FAILED      | 2015-04-06T17:57:16Z |
+-----------------------+------------+--------------------+----------------------+

If the stack list is empty, this indicates an issue with the initial Heat setup. Check your Heat templates and configuration options, and check for any error messages that presented after running openstack overcloud deploy.

Check ironic to see all registered nodes and their current status:

$ source ~/stackrc
(undercloud) $ openstack baremetal node list

+----------+------+---------------+-------------+-----------------+-------------+
| UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
+----------+------+---------------+-------------+-----------------+-------------+
| f1e261...| None | None          | power off   | available       | False       |
| f0b8c1...| None | None          | power off   | available       | False       |
+----------+------+---------------+-------------+-----------------+-------------+

Here are some common issues that arise from the provisioning process.

Many things can occur during the configuration stage. For example, a particular Puppet module could fail to complete due to an issue with the setup. This section provides a process to diagnose such issues.

List all the resources from the overcloud stack to see which one failed:

$ source ~/stackrc
(undercloud) $ openstack stack resource list overcloud --filter status=FAILED

This shows a table of all failed resources.

Show the failed resource:

(undercloud) $ openstack stack resource show overcloud [FAILED RESOURCE]

Check for any information in the resource_status_reason field that can help your diagnosis.

Use the nova command to see the IP addresses of the overcloud nodes.

(undercloud) $ openstack server list

Log in as the heat-admin user to one of the deployed nodes. For example, if the stack’s resource list shows the error occurred on a Controller node, log in to a Controller node. The heat-admin user has sudo access.

(undercloud) $ ssh heat-admin@192.168.24.14

Check the os-collect-config log for a possible reason for the failure.

[heat-admin@overcloud-controller-0 ~]$ sudo journalctl -u os-collect-config

In some cases, nova fails deploying the node in entirety. This situation would be indicated by a failed OS::Heat::ResourceGroup for one of the overcloud role types. Use nova to see the failure in this case.

(undercloud) $ openstack server list
(undercloud) $ openstack server show [SERVER ID]

The most common error shown will reference the error message No valid host was found. See Section 15.6, “Troubleshooting "No Valid Host Found" Errors” for details on troubleshooting this error. In other cases, look at the following log files for further troubleshooting:

The post-deployment process for Controller nodes uses five main steps for the deployment. This includes:

Problems can occur when modifying the overcloud stack through the director. Example of stack modifications include:

Modifying the stack is similar to the process of creating the stack, in that the director checks the availability of the requested number of nodes, provisions additional or removes existing nodes, and then applies the Puppet configuration. Here are some guidelines to follow in situations when modifying the overcloud stack.

As an initial step, follow the advice set in Section 15.4.4, “Post-Deployment Configuration”. These same steps can help diagnose problems with updating the overcloud heat stack. In particular, use the following command to help identify problematic resources:

The next few sections provide advice to diagnose issues on specific node types.

The overcloud Controller nodes contain the bulk of Red Hat OpenStack Platform services. Likewise, you might use multiple Controller nodes in a high availability cluster. If a certain service on a node is faulty, the high availability cluster provides a certain level of failover. However, it then becomes necessary to diagnose the faulty service to ensure your overcloud operates at full capacity.

The Controller nodes use Pacemaker to manage the resources and services in the high availability cluster. The Pacemaker Configuration System (pcs) command is a tool that manages a Pacemaker cluster. Run this command on a Controller node in the cluster to perform configuration and monitoring functions. Here are few commands to help troubleshoot overcloud services on a high availability cluster:

Use these Pacemaker commands to identify the faulty component and/or node. After identifying the component, view the respective component log file in /var/log/.

If a containerized service fails during or after overcloud deployment, use the following recommendations to determine the root cause for the failure:

$ sudo docker logs keystone

In most cases, this log provides the cause of a container’s failure.

$ sudo docker inspect keystone

This provides a JSON object containing low-level configuration data. You can pipe the output to the jq command to parse specific data. For example, to view the container mounts for the keystone container, run the following command:

$ sudo docker inspect keystone | jq .[0].Mounts

You can also use the --format option to parse data to a single line, which is useful for running commands against sets of container data. For example, to recreate the options used to run the keystone container, use the following inspect command with the --format option:

$ sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone

Use these options in conjunction with the docker run command to recreate the container for troubleshooting purposes:

$ OPTIONS=$( sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone )
$ sudo docker run --rm $OPTIONS /bin/bash

$ sudo docker exec -ti keystone <COMMAND>

Replace <COMMAND> with your desired command. For example, each container has a health check script to verify the service connection. You can run the health check script for keystone with the following command:

$ sudo docker exec -ti keystone /openstack/healthcheck

To access the container’s shell, run docker exec using /bin/bash as the command:

$ sudo docker exec -ti keystone /bin/bash

$ sudo docker export keystone -o keystone.tar

This command create the keystone.tar archive, which you can extract and explore.

Compute nodes use the Compute service to perform hypervisor-based operations. This means the main diagnosis for Compute nodes revolves around this service. For example:

For any issues that occur with Red Hat Ceph Storage clusters, see "Logging Configuration Reference" in the Red Hat Ceph Storage Configuration Guide. This section provides information on diagnosing logs for all Ceph storage services.