Red Hat SummitEste conteúdo não está disponível no idioma selecionado.
Chapter 8. Troubleshooting Director-Based Upgrades
This section provides advice for troubleshooting issues with both.
8.1. Undercloud Upgrades
In situations where an Undercloud upgrade command (openstack undercloud upgrade) fails, use the following advice to locate the issue blocking upgrade progress:
-
The
openstack undercloud upgradecommand prints out a progress log while it runs. If an error occurs at any point in the upgrade process, the command halts at the point of error. Use this information to identify any issues impeding upgrade progress. The
openstack undercloud upgradecommand runs Puppet to configure Undercloud services. This generates useful Puppet reports in the following directories:-
/var/lib/puppet/state/last_run_report.yaml- The last Puppet reports generated for the Undercloud. This file shows any causes of failed Puppet actions. -
/var/lib/puppet/state/last_run_summary.yaml- A summary of thelast_run_report.yamlfile. /var/lib/puppet/reports- All Puppet reports for the Undercloud.Use this information to identify any issues impeding upgrade progress.
-
Check for any failed services:
sudo systemctl -t service
$ sudo systemctl -t serviceCopy to Clipboard Copied! Toggle word wrap Toggle overflow If any services have failed, check their corresponding logs. For example, if
openstack-ironic-apifailed, use the following commands to check the logs for that service:sudo journalctl -xe -u openstack-ironic-api sudo tail -n 50 /var/log/ironic/ironic-api.log
$ sudo journalctl -xe -u openstack-ironic-api $ sudo tail -n 50 /var/log/ironic/ironic-api.logCopy to Clipboard Copied! Toggle word wrap Toggle overflow
After correcting the issue impeding the Undercloud upgrade, rerun the upgrade command:
openstack undercloud upgrade
$ openstack undercloud upgradeThe upgrade command begins again and configures the Undercloud.
8.2. Overcloud Upgrades
In situations where an Overcloud upgrade process fails, use the following advice to locate the issue blocking upgrade progress:
Check the Heat stack listing and identify any stacks that have an
UPDATE_FAILEDstatus. The following command identifies these stacks:heat stack-list --show-nested | awk -F "|" '{ print $3,$4 }' | grep "UPDATE_FAILED" | column -t$ heat stack-list --show-nested | awk -F "|" '{ print $3,$4 }' | grep "UPDATE_FAILED" | column -tCopy to Clipboard Copied! Toggle word wrap Toggle overflow View the failed stack and its template to identify how the stack failed:
heat stack-show overcloud-Controller-qyoy54dyhrll-1-gtwy5bgta3np heat template-show overcloud-Controller-qyoy54dyhrll-1-gtwy5bgta3np
$ heat stack-show overcloud-Controller-qyoy54dyhrll-1-gtwy5bgta3np $ heat template-show overcloud-Controller-qyoy54dyhrll-1-gtwy5bgta3npCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check that Pacemaker is running correctly on all Controller nodes. If necessary, log into a Controller node and restart the Controller cluster:
sudo pcs cluster start
$ sudo pcs cluster startCopy to Clipboard Copied! Toggle word wrap Toggle overflow
After correcting the issue impeding the Overcloud upgrade, rerun the openstack overcloud deploy command for the failed upgrade step you attempted. This following is an example of the first openstack overcloud deploy command in the upgrade process, which includes the major-upgrade-pacemaker-init.yaml:
openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \ -e network_env.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
$ openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
-e network_env.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
The openstack overcloud deploy retries the Overcloud stack update.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}