Chapter 14. Troubleshooting director errors

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the node import command with the --validate-only option. This option validates your node template without performing an import:

   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json
   Waiting for messages on queue 'tripleo' with no timeout.
   
   Successfully validated environment file

3. To fix incorrect details with imported nodes, run the openstack baremetal commands to update node details. The following example shows how to change networking details:

   1. Identify the assigned port UUID for the imported node:

      $ source ~/stackrc
      (undercloud) $ openstack baremetal port list --node [NODE UUID]

   2. Update the MAC address:

      (undercloud) $ openstack baremetal port set --address=[NEW MAC] [PORT UUID]

   3. Configure a new IPMI address on the node:

      (undercloud) $ openstack baremetal node set --driver-info ipmi_address=[NEW IPMI ADDRESS] [NODE UUID]

Procedure

1. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

2. Ensure that your nodes are in a manageable state. The introspection does not inspect nodes in an available state, which is meant for deployment. If you want to inspect nodes that are in an available state, change the node status to manageable state before introspection:

   (undercloud)$ openstack baremetal node manage <node_uuid>

3. To configure temporary access to the introspection RAM disk during introspection debugging, use the sshkey parameter to append your public SSH key to the kernel configuration in the /httpboot/inspector.ipxe file:

   kernel http://192.2.0.1:8088/agent.kernel ipa-inspection-callback-url=http://192.168.0.1:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=${mac} ipa-debug=1 ipa-inspection-benchmarks=cpu,mem,disk selinux=0 sshkey="<public_ssh_key>"

4. Run the introspection on the node:

   (undercloud)$ openstack overcloud node introspect <node_uuid> --provide

   Use the --provide option to change the node state to available after the introspection completes.

5. Identify the IP address of the node from the dnsmasq logs:

   (undercloud)$ sudo tail -f /var/log/containers/ironic-inspector/dnsmasq.log

6. If an error occurs, access the node using the root user and temporary access details:

   $ ssh root@192.168.24.105

   Access the node during introspection to run diagnostic commands and troubleshoot the introspection failure.

7. To stop the introspection process, run the following command:

   (undercloud)$ openstack baremetal introspection abort <node_uuid>

   You can also wait until the process times out.

   Note

   Red Hat OpenStack Platform director retries introspection three times after the initial abort. Run the openstack baremetal introspection abort command at each attempt to abort the introspection completely.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Launch the ephemeral Heat process:

   (undercloud)$ openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/overcloud/heat-launcher --restore-db
   (undercloud)$ export OS_CLOUD=heat

3. View the details of the failure:

   (undercloud)$ openstack stack failures list <overcloud> --long

   • Replace <overcloud> with the name of your overcloud.

4. Identify the stacks that failed:

   (undercloud)$ openstack stack list --nested --property status=FAILED

5. Remove the ephemeral Heat process from the undercloud:

   (undercloud)$ openstack tripleo launch heat --kill

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal service to see all registered nodes and their current status:

   (undercloud) $ openstack baremetal node list
   
   +----------+------+---------------+-------------+-----------------+-------------+
   | UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
   +----------+------+---------------+-------------+-----------------+-------------+
   | f1e261...| None | None          | power off   | available       | False       |
   | f0b8c1...| None | None          | power off   | available       | False       |
   +----------+------+---------------+-------------+-----------------+-------------+

   All nodes available for provisioning should have the following states set:

   • Maintenance set to False.
   • Provision State set to available before provisioning.

3. If a node does not have Maintenance set to False or Provision State set to available, then use the following table to identify the problem and the solution:

   Expand

   Problem	Cause	Solution
   Maintenance sets itself to True automatically.	Director cannot access the power management for the nodes.	Check the credentials for node power management.
   Provision State is set to available but nodes do not provision.	The problem occurred before bare-metal deployment started.	Check that the node hardware details are within the requirements.
   Provision State is set to wait call-back for a node.	The node provisioning process has not yet finished for this node.	Wait until this status changes. Otherwise, connect to the virtual console of the node and check the output.
   Provision State is active and Power State is power on but the nodes do not respond.	The node provisioning has finished successfully and there is a problem during the post-deployment configuration step.	Diagnose the node configuration process. Connect to the virtual console of the node and check the output.
   Provision State is error or deploy failed.	Node provisioning has failed.	View the bare metal node details with the openstack baremetal node show command and check the last_error field, which contains error description.

   Show more

Procedure

1. Install nmap:

   $ sudo dnf install nmap

2. Use nmap to scan the IP address range for active addresses. This example scans the 192.168.24.0/24 range, replace this with the IP subnet of the Provisioning network (using CIDR bitmask notation):

   $ sudo nmap -sn 192.168.24.0/24

3. Review the output of the nmap scan. For example, you should see the IP address of the undercloud, and any other hosts that are present on the subnet:

   $ sudo nmap -sn 192.168.24.0/24
   
   Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-02 15:14 EDT
   Nmap scan report for 192.168.24.1
   Host is up (0.00057s latency).
   Nmap scan report for 192.168.24.2
   Host is up (0.00048s latency).
   Nmap scan report for 192.168.24.3
   Host is up (0.00045s latency).
   Nmap scan report for 192.168.24.5
   Host is up (0.00040s latency).
   Nmap scan report for 192.168.24.9
   Host is up (0.00019s latency).
   Nmap done: 256 IP addresses (5 hosts up) scanned in 2.45 seconds

   If any of the active IP addresses conflict with the IP ranges in undercloud.conf, you must either change the IP address ranges or release the IP addresses before you introspect or deploy the overcloud nodes.

Procedure

1. Ensure that the stack user has access to the files in the ~/config-download/overcloud directory on the undercloud:

   $ sudo setfacl -R -m u:stack:rwx ~/config-download/overcloud

2. Change to the working directory for the config-download files:

   $ cd ~/config-download/overcloud

3. Search the ansible.log file for the point of failure:

   $ less ansible.log

   Make a note of the step that failed.

4. Find the step that failed in the config-download playbooks within the working directory to identify the action that occurred.

Accessing the host

1. Source the stackrc file:

   $ source ~/stackrc

2. Get the IP address of the node with the container failure.

   (undercloud) $ metalsmith list

3. Log in to the node:

   (undercloud) $ ssh tripleo-admin@192.168.24.60

Identifying failed containers

1. View all containers:

   $ sudo podman ps --all

   Identify the failed container. The failed container usually exits with a non-zero status.

Checking container logs

1. Each container retains standard output from its main process. Use this output as a log to help determine what actually occurs during a container run. For example, to view the log for the keystone container, run the following command:

   $ sudo podman logs keystone

   In most cases, this log contains information about the cause of a container failure.

2. The host also retains the stdout log for the failed service. You can find the stdout logs in /var/log/containers/stdouts/. For example, to view the log for a failed keystone container, run the following command:

   $ cat /var/log/containers/stdouts/keystone.log

Viewing a container filesystem

1. To view the file system for the failed container, run the podman mount command. For example, to view the file system for a failed keystone container, run the following command:

   $ sudo podman mount keystone

   This command provides a mounted location to view the file system contents:

   /var/lib/containers/storage/overlay/78946a109085aeb8b3a350fc20bd8049a08918d74f573396d7358270e711c610/merged

   This output is useful for viewing the Puppet reports within the container. You can find these reports in the var/lib/puppet/ directory within the container mount.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Get the IP address of the Compute node that contains the failure:

   (undercloud) $ openstack server list

3. Log in to the node:

   (undercloud) $ ssh tripleo-admin@192.168.24.60

4. Change to the root user:

   $ sudo -i

5. View the status of the container:

   $ sudo podman ps -f name=nova_compute

6. The primary log file for Compute nodes is /var/log/containers/nova/nova-compute.log. If issues occur with Compute node communication, use this file to begin the diagnosis.
7. If you perform maintenance on the Compute node, migrate the existing instances from the host to an operational Compute node, then disable the node.