Chapter 9. Troubleshooting
The in-place upgrade is a complex process, and it is common to encounter issues and roadblocks. Refer to the following troubleshooting resources and tips for help on resolving these issues.
9.1. Troubleshooting resources Copy linkLink copied to clipboard!
You can use various troubleshooting resources can help you diagnose and troubleshoot issues you encounter throughout the upgrade process.
Console output
By default, only error and critical log level messages are printed to the console output by the Leapp utility. To change the log level, use the --verbose or --debug options with the leapp upgrade command.
-
In verbose mode,
Leappprints info, warning, error, and critical messages. -
In debug mode,
Leappprints debug, info, warning, error, and critical messages.
Logs
-
The
/var/log/leapp/leapp-upgrade.logfile lists issues found during the initramfs phase. -
The
/var/log/leapp/dnf-debugdata/directory contains transaction debug data. This directory is present only if theleapp upgradecommand is executed with the--debugoption. -
The
/var/log/leapp/answerfilecontains questions required to be answered byLeapp. -
The
journalctlutility provides complete logs.
Reports
-
The
/var/log/leapp/leapp-report.txtfile lists issues found during the pre-upgrade phase. The report is also available in the web console, see Assessing upgradability and applying automated remediations through the web console. -
The
/var/log/leapp/leapp-report.jsonfile lists issues found during the pre-upgrade phase in a machine-readable format, which enables you to process the report using custom scripts. For more information, see Automating your Red Hat Enterprise Linux pre-upgrade report workflow.
9.2. Troubleshooting tips Copy linkLink copied to clipboard!
When diagnosing and troubleshooting issues that occur during the in-place upgrade process, make sure to check for these frequently skipped steps and use these helpful resources.
Pre-upgrade phase
- Verify that your system meets all conditions listed in Planning an upgrade.
-
Make sure you have followed all steps described in Preparing for the upgrade, for example, your system does not use more than one Network Interface Card (NIC) with a name based on the prefix used by the kernel (
eth). Make sure you have answered all questions required by
Leappin the/var/log/leapp/answerfilefile. If any answers are missing,Leappinhibits the upgrade. For example:- Are there no VDO devices on the system?
-
Make sure you have resolved all problems identified in the pre-upgrade report, located at
/var/log/leapp/leapp-report.txt. To achieve this, you can also use the web console, as described in Assessing upgradability and applying automated remediations through the web console.
Example 9.1. Leapp answerfile
The following is an example of an unedited /var/log/leapp/answerfile file that has one unanswered question:
The Label field specifies the question that requires an answer. In this example, the question is Are all VDO devices, if any, successfully converted to LVM management?
To answer the question, uncomment the last line and enter an answer of True or False. In this example, the selected answer is True:
[check_vdo] ... # Available choices: True/False # Unanswered question. Uncomment the following line with your answer confirm = True
[check_vdo]
...
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True
Download phase
If a problem occurs during downloading RPM packages, examine transaction debug data located in the
/var/log/leapp/dnf-debugdata/directory.NoteThe
/var/log/leapp/dnf-debugdata/directory is empty or does not exist if no transaction debug data was produced. This might occur when the required repositories are not available.
Initramfs phase
During this phase, potential failures redirect you to the Dracut shell. Check the Journal log:
journalctl
# journalctlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Alternatively, restart the system from the Dracut shell using the
rebootcommand and check the/var/log/leapp/leapp-upgrade.logfile.
Post-upgrade phase
- If your system seems to be successfully upgraded but booted with the old RHEL 9 kernel, restart the system and check the kernel version of the default entry in GRUB.
- Make sure you have followed the recommended steps in Verifying the post-upgrade state.
If your application or a service stops working or behaves incorrectly after you have switched SELinux to enforcing mode, search for denials using the
ausearch,journalctl, ordmesgutilities:ausearch -m AVC,USER_AVC -ts boot journalctl -t setroubleshoot dmesg | grep -i -e selinux -e type=1400
# ausearch -m AVC,USER_AVC -ts boot # journalctl -t setroubleshoot # dmesg | grep -i -e selinux -e type=1400Copy to Clipboard Copied! Toggle word wrap Toggle overflow The most common problems are caused by incorrect labeling. See Troubleshooting problems related to SELinux for more details.
9.3. Known issues for the RHEL 9.7 to RHEL 10.1 upgrade Copy linkLink copied to clipboard!
There are a variety of known issues that you might encounter when upgrading from RHEL 9.7 to RHEL 10.1.
-
If your RHEL 9 system uses a device driver that is provided by Red Hat but is not available in RHEL 10,
Leappinhibits the upgrade. However, if the RHEL 9 system uses a third-party device driver thatLeappdoes not have data for in the/etc/leapp/files/device_driver_deprecation_data.jsonfile,Leappdoes not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade. If the name of a third-party package, not signed by Red Hat, installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, choose one of the following options prior to upgrading:
- Remove the third-party package
- Replace the third-party package with the package provided by Red Hat
- The in-place upgrade might fail on systems with Software Redundant Array of Independent Disks (RAID). (BZ#1957192)
During the in-place upgrade, the
Leapputility usually preserves the network interface controller (NIC) names between RHEL 9 and RHEL 10. However, on some systems, such as systems with network bonding, the NIC names might need to be updated between RHEL 9 and RHEL 10. On those systems, perform the following steps:-
Set the
LEAPP_NO_NETWORK_RENAMING=1environment variable to prevent the Leapp utility from incorrectly preserving the original RHEL 9 NIC names. - Perform the in-place upgrade.
Verify that your network is working correctly. If needed, manually update the network configuration.
(BZ#1919382)
-
Set the
If any of the mounted file systems that are defined in the
/etc/fstabfile do not have thesharedpropagation flag set, the upgrade might fail. To prevent this issue, remount these file systems to set them as shared:mount -o remount --make-shared <mountpoint>
# mount -o remount --make-shared <mountpoint>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace mountpoint with the mountpoint of each file system.
For more information, see the Red Hat Knowledgebase solution Leapp "Can not load RPM file" during the DNF transaction check. (RHEL-23449)
-
If you use an HTTP proxy, you must configure Red Hat Subscription Manager (RHSM) to use such a proxy, or you must execute the
subscription-managercommand with the--proxy <hostname>option. Otherwise, an execution of thesubscription-managercommand fails. If you use the--proxyoption instead of the configuration change, the upgrade process fails becauseLeappis unable to detect the proxy. To prevent this problem from occurring, manually edit therhsm.conffile. For more information, see the Red Hat Knowledgebase solution How to configure HTTP Proxy for Red Hat Subscription Management. (BZ#1689294) -
For systems that require a proxy to access RHEL 9 content, you usually need to configure the use of the proxy by DNF in the
/etc/dnf/dnf.confconfiguration file. If the current DNF configuration is not compatible with the DNF version on the target system, specify the valid target configuration in the/etc/leapp/files/dnf.confconfiguration file. For more information, see the Red Hat Knowledgebase solution How does Leapp work with a proxy? -
The kerberos client might break after the upgrade if it is configured to use the deprecated
/etc/ssl/certs/ca-certificates.crtfile for root certificates. To fix the configuration, use the/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemfile instead. (RHEL-65265) - On IBM Z machines, the upgrade might fail if the system is on multipath LVM SCSI LUNs. (RHEL-76159)
-
The upgrade might fail if you are upgrading by using Red Hat Update Infrastructure (RHUI) with an ISO image. You can work around this issue by not using the
--isooption with the upgrade or see the Red Hat Knowledgebase solution Offline Leapp upgrade using ISO fails with "Failed to synchronize cache for repo 'rhul-microsoft-azure-rhel8', ignoring this repo. (RHEL-3296) -
If you are upgrading by using Red Hat Update Infrastructure (RHUI), files in the
/usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/directory are incorrectly reported as custom files in the pre-upgrade report. Unless you modified these files manually, you can ignore the warnings about these files in the report and the in-place upgrade will not be affected. (RHEL-40115) -
When upgrading systems by using Red Hat Upgrade Infrastructure (RHUI), the upgrade could fail if the RHUI setup of the system differs from defaults implemented in the in-place upgrade solution RHUI systems expected by the
Leapputility. To resolve this problem, configure the upgrade process to adjust it for your RHUI setup. For more information, see Using RHUI to configure an in-place upgrade.
9.4. Known issues for the RHEL 9.6 to RHEL 10.0 upgrade Copy linkLink copied to clipboard!
There are a variety of known issues that you might encounter when upgrading from RHEL 9.6 to RHEL 10.1.
-
If your RHEL 9 system uses a device driver that is provided by Red Hat but is not available in RHEL 10,
Leappinhibits the upgrade. However, if the RHEL 9 system uses a third-party device driver thatLeappdoes not have data for in the/etc/leapp/files/device_driver_deprecation_data.jsonfile,Leappdoes not detect such a driver and proceeds with the upgrade. Consequently, the system might fail to boot after the upgrade. If the name of a third-party package, not signed by Red Hat, installed on your system is the same as the name of a package provided by Red Hat, the in-place upgrade fails. To work around this problem, choose one of the following options prior to upgrading:
- Remove the third-party package
- Replace the third-party package with the package provided by Red Hat
- The in-place upgrade might fail on systems with Software Redundant Array of Independent Disks (RAID). (BZ#1957192)
During the in-place upgrade, the
Leapputility usually preserves the network interface controller (NIC) names between RHEL 9 and RHEL 10. However, on some systems, such as systems with network bonding, the NIC names might need to be updated between RHEL 9 and RHEL 10. On those systems, perform the following steps:-
Set the
LEAPP_NO_NETWORK_RENAMING=1environment variable to prevent the Leapp utility from incorrectly preserving the original RHEL 9 NIC names. - Perform the in-place upgrade.
Verify that your network is working correctly. If needed, manually update the network configuration.
(BZ#1919382)
-
Set the
If any of the mounted file systems that are defined in the
/etc/fstabfile do not have thesharedpropagation flag set, the upgrade might fail. To prevent this issue, remount these file systems to set them as shared:mount -o remount --make-shared <mountpoint>
# mount -o remount --make-shared <mountpoint>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace mountpoint with the mountpoint of each file system.
For more information, see the Red Hat Knowledgebase solution Leapp "Can not load RPM file" during the DNF transaction check. (RHEL-23449)
-
If you use an HTTP proxy, Red Hat Subscription Manager must be configured to use such a proxy, or the
subscription-managercommand must be executed with the--proxy <hostname>option. Otherwise, an execution of thesubscription-managercommand fails. If you use the--proxyoption instead of the configuration change, the upgrade process fails becauseLeappis unable to detect the proxy. To prevent this problem from occurring, manually edit therhsm.conffile. For more information, see the Red Hat Knowledgebase solution How to configure HTTP Proxy for Red Hat Subscription Management. (BZ#1689294) -
For systems that require a proxy to access RHEL 9 content, you usually need to configure the use of the proxy by DNF in the
/etc/dnf/dnf.confconfiguration file. If the current DNF configuration is not compatible with the DNF version on the target system, specify the valid target configuration in the/etc/leapp/files/dnf.confconfiguration file. For more information, see the Red Hat Knowledgebase solution How does Leapp work with a proxy?
-
The kerberos client might break after the upgrade if it is configured to use the deprecated
/etc/ssl/certs/ca-certificates.crtfile for root certificates. To fix the configuration, use the/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemfile instead. (RHEL-65265) - On IBM Z machines, the upgrade might fail if the system is on multipath LVM SCSI LUNs. (RHEL-76159)
If
dracutis configured to include the deprecatednetwork-legacymodule, the system does not boot after the upgrade. This issue often occurs on systems that have been in-place upgraded to RHEL 9. To prevent this problem, perform the following actions:-
Remove the
network-legacymodule from thedracutconfiguration. - Rebuild existing initramfs images.
Reboot the system before you start the upgrade.
For more information see the leapp upgrade fails to boot after upgrading to RHEL 10.0 Knowledgebase solution.
-
Remove the
9.5. Obtaining support Copy linkLink copied to clipboard!
To open a support case, select RHEL 9 as the product, and provide a sosreport from your system.
-
To generate a
sosreporton your system, run:
sosreport
# sosreport
Note that you can leave the case ID empty.
For more information about generating a sosreport, see the Red Hat Knowledgebase solution What is an sosreport and how to create one in Red Hat Enterprise Linux?.
For more information about opening and managing a support case on the Customer Portal, see the Red Hat Knowledgebase solution, How do I open and manage a support case on the Customer Portal?.