Chapter 22. Troubleshooting director errors

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the node import command with the --validate-only option. This option validates your node template without performing an import:

   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json
   Waiting for messages on queue 'tripleo' with no timeout.
   
   Successfully validated environment file

   Copy to Clipboard Toggle word wrap

3. To fix incorrect details with imported nodes, run the openstack baremetal commands to update node details. The following example shows how to change networking details:

   1. Identify the assigned port UUID for the imported node:

      $ source ~/stackrc
      (undercloud) $ openstack baremetal port list --node [NODE UUID]

      Copy to Clipboard Toggle word wrap

   2. Update the MAC address:

      (undercloud) $ openstack baremetal port set --address=[NEW MAC] [PORT UUID]

      Copy to Clipboard Toggle word wrap

   3. Configure a new IPMI address on the node:

      (undercloud) $ openstack baremetal node set --driver-info ipmi_address=[NEW IPMI ADDRESS] [NODE UUID]

      Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. The director uses OpenStack Object Storage (swift) to save the hardware data obtained during the introspection process. If this service is not running, the introspection can fail. Check all services related to OpenStack Object Storage to ensure the service is running:

   (undercloud) $ sudo systemctl list-units tripleo_swift*

   Copy to Clipboard Toggle word wrap

3. Check your nodes are in a manageable state. The introspection does not inspect nodes in an available state, which is meant for deployment. In this situation, change the node status to manageable state before introspection:

   (undercloud) $ openstack baremetal node manage [NODE UUID]

   Copy to Clipboard Toggle word wrap

4. Configure temporary access to the introspection ramdisk. You can provide either a temporary password or an SSH key to access the node during introspection debugging. Complete the following procedure to configure ramdisk access:

   1. Run the openssl passwd -1 command with a temporary password to generate an MD5 hash:

      (undercloud) $ openssl passwd -1 mytestpassword
      $1$enjRSyIw$/fYUpJwr6abFy/d.koRgQ/

      Copy to Clipboard Toggle word wrap

   2. Edit the /var/lib/ironic/httpboot/inspector.ipxe file, find the line starting with kernel, and append the rootpwd parameter and the MD5 hash. For example:

      kernel http://192.2.0.1:8088/agent.kernel ipa-inspection-callback-url=http://192.168.0.1:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=${mac} ipa-debug=1 ipa-inspection-benchmarks=cpu,mem,disk rootpwd="$1$enjRSyIw$/fYUpJwr6abFy/d.koRgQ/" selinux=0

      Copy to Clipboard Toggle word wrap

      Alternatively, append your public SSH key to the sshkey parameter.

      Note

      Include quotation marks for both the rootpwd and sshkey parameters.

5. Run the introspection on the node:

   (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

   Copy to Clipboard Toggle word wrap

   The --provide option causes the node state to change to available when the introspection completes.

6. Identify the IP address of the node from the dnsmasq logs:

   (undercloud) $ sudo tail -f /var/log/containers/ironic-inspector/dnsmasq.log

   Copy to Clipboard Toggle word wrap

7. If an error occurs, access the node using the root user and temporary access details:

   $ ssh root@192.168.24.105

   Copy to Clipboard Toggle word wrap

   Accessing the node during introspection means you can run diagnostic commands to troubleshoot the introspection failure.

8. To stop the introspection process, run the following command:

   (undercloud) $ openstack baremetal introspection abort [NODE UUID]

   Copy to Clipboard Toggle word wrap

   You can also wait until the process times out.

   Note

   OpenStack Platform director retries introspection three times after the initial abort. Run the openstack baremetal introspection abort command at each attempt to abort the introspection completely.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. List the workflow executions that have the failed state ERROR:

   (undercloud) $ openstack workflow execution list | grep "ERROR"

   Copy to Clipboard Toggle word wrap

3. Get the UUID of the failed workflow execution (for example, dffa96b0-f679-4cd2-a490-4769a3825262) and view the execution and its output:

   (undercloud) $ openstack workflow execution show dffa96b0-f679-4cd2-a490-4769a3825262
   (undercloud) $ openstack workflow execution output show dffa96b0-f679-4cd2-a490-4769a3825262

   Copy to Clipboard Toggle word wrap

4. These commands return information about the failed task in the execution. The openstack workflow execution show command also displays the workflow used for the execution (for example, tripleo.plan_management.v1.publish_ui_logs_to_swift). You can view the full workflow definition using the following command:

   (undercloud) $ openstack workflow definition show tripleo.plan_management.v1.publish_ui_logs_to_swift

   Copy to Clipboard Toggle word wrap

   This is useful for identifying where in the workflow a particular task occurs.

5. View action executions and their results using a similar command syntax:

   (undercloud) $ openstack action execution list
   (undercloud) $ openstack action execution show 8a68eba3-0fec-4b2a-adc9-5561b007e886
   (undercloud) $ openstack action execution output show 8a68eba3-0fec-4b2a-adc9-5561b007e886

   Copy to Clipboard Toggle word wrap

   This is useful for identifying a specific action that causes issues.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Run the deployment failures command:

   $ openstack overcloud failures

   Copy to Clipboard Toggle word wrap

3. Run the following command to display details of the failure:

   (undercloud) $ openstack stack failures list <OVERCLOUD_NAME> --long

   Copy to Clipboard Toggle word wrap

   Replace <OVERCLOUD_NAME> with the name of your overcloud.

4. Run the following command to identify the stacks that failed:

   (undercloud) $ openstack stack list --nested --property status=FAILED

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check the bare metal service to see all registered nodes and their current status:

   (undercloud) $ openstack baremetal node list
   
   +----------+------+---------------+-------------+-----------------+-------------+
   | UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
   +----------+------+---------------+-------------+-----------------+-------------+
   | f1e261...| None | None          | power off   | available       | False       |
   | f0b8c1...| None | None          | power off   | available       | False       |
   +----------+------+---------------+-------------+-----------------+-------------+

   Copy to Clipboard Toggle word wrap

   All nodes available for provisioning should have the following states set:

   • Maintenance set to False.
   • Provision State set to available before provisioning.

   The following table outlines some common provisioning failure scenarios.

Procedure

1. Install nmap:

   $ sudo dnf install nmap

   Copy to Clipboard Toggle word wrap

2. Use nmap to scan the IP address range for active addresses. This example scans the 192.168.24.0/24 range, replace this with the IP subnet of the Provisioning network (using CIDR bitmask notation):

   $ sudo nmap -sn 192.168.24.0/24

   Copy to Clipboard Toggle word wrap

3. Review the output of the nmap scan. For example, you should see the IP address of the undercloud, and any other hosts that are present on the subnet:

   $ sudo nmap -sn 192.168.24.0/24
   
   Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-02 15:14 EDT
   Nmap scan report for 192.168.24.1
   Host is up (0.00057s latency).
   Nmap scan report for 192.168.24.2
   Host is up (0.00048s latency).
   Nmap scan report for 192.168.24.3
   Host is up (0.00045s latency).
   Nmap scan report for 192.168.24.5
   Host is up (0.00040s latency).
   Nmap scan report for 192.168.24.9
   Host is up (0.00019s latency).
   Nmap done: 256 IP addresses (5 hosts up) scanned in 2.45 seconds

   Copy to Clipboard Toggle word wrap

   If any of the active IP addresses conflict with the IP ranges in undercloud.conf, you will need to either change the IP address ranges or free up the IP addresses before introspecting or deploying the overcloud nodes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Check that the introspection succeeded on the node. If the introspection fails, check that each node contains the required ironic node properties:

   (undercloud) $ openstack baremetal node show [NODE UUID]

   Copy to Clipboard Toggle word wrap

   Check the properties JSON field has valid values for keys cpus, cpu_arch, memory_mb and local_gb.

3. Check the Compute flavor mapped to the node:

   (undercloud) $ openstack flavor show [FLAVOR NAME]

   Copy to Clipboard Toggle word wrap

   Make sure it does not exceed the node properties for the required number of nodes.

4. Run the openstack baremetal node list command to ensure sufficient nodes in the available state. Nodes in manageable state usually signify a failed introspection.

5. Run the openstack baremetal node list command to check the nodes are not in maintenance mode. If a node changes to maintenance mode automatically, the likely cause is an issue with incorrect power management credentials. Check the power management credentials and then remove maintenance mode:

   (undercloud) $ openstack baremetal node maintenance unset [NODE UUID]

   Copy to Clipboard Toggle word wrap
6. If you are using automatic profile tagging, check that you have enough nodes corresponding to each flavor and profile. Run the openstack baremetal node show command on a node and check the capabilities key in the properties field. For example, a node tagged for the Compute role should contain profile:compute.

7. It takes some time for node information to propagate from Bare Metal to Compute after introspection. However, if you performed some steps manually, there might be a short period of time when nodes are not available to nova. Use the following command to check the total resources in your system:

   (undercloud) $ openstack hypervisor stats show

   Copy to Clipboard Toggle word wrap

Procedure

1. Make sure the stack user has access to the files in the /var/lib/mistral directory on the undercloud:

   $ sudo setfacl -R -m u:stack:rwx /var/lib/mistral

   Copy to Clipboard Toggle word wrap

   This command retains mistral user access to the directory.

2. Change to the working directory for the config-download files. This is usually /var/lib/mistral/overcloud/.

   $ cd /var/lib/mistral/overcloud/

   Copy to Clipboard Toggle word wrap

3. Search the ansible.log file for the point of failure.

   $ less ansible.log

   Copy to Clipboard Toggle word wrap

   Make a note of the step that failed.

4. Find the step that failed in the config-download playbooks within the working directory to identify the action that took place.

Accessing the host

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Get the IP address of the node with the container failure.

   (undercloud) $ openstack server list

   Copy to Clipboard Toggle word wrap

3. Log into the node:

   (undercloud) $ ssh heat-admin@192.168.24.60

   Copy to Clipboard Toggle word wrap

4. Change to the root user:

   $ sudo -i

   Copy to Clipboard Toggle word wrap

Identifying failed containers

1. View all containers:

   $ podman ps --all

   Copy to Clipboard Toggle word wrap

   Identify the failed container. The failed container usually exits with a non-zero status.

Checking container logs

1. Each container retains standard output from its main process. Use this output as a log to help determine what actually occurs during a container run. For example, to view the log for the keystone container, use the following command:

   $ sudo podman logs keystone

   Copy to Clipboard Toggle word wrap

   In most cases, this log contains information about the cause of a container’s failure.

2. The host also retains the stdout log for the failed service. You can find the stdout logs in /var/log/containers/stdouts/. For example, to view the log for a failed keystone container, run the following command:

   $ cat /var/log/containers/stdouts/keystone.log

   Copy to Clipboard Toggle word wrap

Viewing a container filesystem

1. To view the file system for the failed container, run the podman mount command. For example, to view the file system for a failed keystone container, run the following command:

   $ podman mount keystone

   Copy to Clipboard Toggle word wrap

   This provides a mounted location to view the filesystem contents:

   /var/lib/containers/storage/overlay/78946a109085aeb8b3a350fc20bd8049a08918d74f573396d7358270e711c610/merged

   Copy to Clipboard Toggle word wrap

   This is useful for viewing the Puppet reports within the container. You can find these reports in the var/lib/puppet/ directory within the container mount.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Get the IP address of the Compute node containing the failure:

   (undercloud) $ openstack server list

   Copy to Clipboard Toggle word wrap

3. Log into the node:

   (undercloud) $ ssh heat-admin@192.168.24.60

   Copy to Clipboard Toggle word wrap

4. Change to the root user:

   $ sudo -i

   Copy to Clipboard Toggle word wrap

5. View the status of the container:

   $ sudo podman ps -f name=nova_compute

   Copy to Clipboard Toggle word wrap
6. The primary log file for Compute nodes is /var/log/containers/nova/nova-compute.log. If issues occur with Compute node communication, this log file is usually a good place to start a diagnosis.
7. If performing maintenance on the Compute node, migrate the existing instances from the host to an operational Compute node, then disable the node.