Upgrading from RHEL 9 to RHEL 10

Procedure

1. Optional: Unmount non-system OS file systems that are not required for the upgrade and comment them out from the /etc/fstab file. For example, this includes file systems containing only data files unrelated to the system itself. This can reduce the amount of time needed for the upgrade process and prevent potential issues related to third-party applications that are not migrated properly during the upgrade by custom or third-party actors.

2. If you are upgrading by using RHSM, verify that the system is registered to an account with Simple Content Access (SCA) enabled:

   Copy to Clipboard Toggle word wrap

   # subscription-manager status
   +-------------------------------------------+
      System Status Details
   +-------------------------------------------+
   Overall Status: Disabled
   Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.
   System Purpose Status: Disabled

3. Ensure you have appropriate repositories enabled. The following command enables the Base and AppStream repositories for the 64-bit Intel architecture; for other architectures, see RHEL 9 repositories.

   Copy to Clipboard Toggle word wrap

   # subscription-manager repos --enable rhel-9-for-x86_64-baseos-rpms --enable rhel-9-for-x86_64-appstream-rpms

   Note

   Optional: Enable the CodeReady Linux Builder (also known as Optional) or Supplementary repositories. For more information about the content of these repositories, see the Package manifest.

4. Set the system release version:

   Copy to Clipboard Toggle word wrap

   # subscription-manager release --set 9.6

5. If you use the dnf versionlock plugin to lock packages to a specific version, clear the lock by running:

   Copy to Clipboard Toggle word wrap

   # dnf versionlock clear

6. Ensure that you have up-to-date leapp and leapp-repository packages:

   1. RHEL 9.6: version 0.19.0 of the leapp package and version 0.22.0 of the leapp-repository package.

      The leapp-repository package contains the leapp-upgrade-el9toel10 RPM package.

      Note

      Disconnected systems only:download the following packages from the Red Hat Customer Portal:

      • leapp
      • leapp-deps
      • python3-leapp
      • leapp-upgrade-el9toel10
      • leapp-upgrade-el9toel10-deps

7. Install the Leapp utility:

   Copy to Clipboard Toggle word wrap

   # dnf install leapp-upgrade

8. Update all packages to the latest RHEL 9 version and reboot:

   Copy to Clipboard Toggle word wrap

   # dnf update
   # reboot

9. Optional: Review, remediate, and then remove the rpmnew and rpmsave files.

10. If you use a configuration management system, ensure that it does not interfere with the in-place upgrade process:

    • If your configuration management system has a client-server architecture, such as Puppet, Salt, or Chef, disable the system before running the leapp preupgrade command. Do not enable the configuration management system until after the upgrade is complete to prevent issues during the upgrade.

    • If your configuration management system has agentless architecture, do not execute the configuration and deployment file. For example, if your system has Ansible, do not execute an Ansible playbook during the upgrade.

      Warning

      Automation of the pre-upgrade and upgrade process by using a configuration management system is not supported by Red Hat. For more information, see Using configuration management systems to automate parts of the Leapp pre-upgrade and upgrade process on Red Hat Enterprise Linux.

11. If you are upgrading by using an ISO image, verify that the ISO image contains the target OS version, for example, RHEL 10.0, and is saved to a persistent local mount point to ensure that the Leapp utility can access the image throughout the upgrade process.

Procedure

1. Import a subscription manifest with RHEL 9 repositories into Satellite Server. For more information, see the Managing Red Hat Subscriptions chapter in the Managing Content Guide for the particular version of Red Hat Satellite.

2. Enable and synchronize all required RHEL 9 and RHEL 10 repositories on the Satellite Server with the latest updates for the source and target OS versions. Required repositories must be available in the Content View and enabled in the associated activation key.

   Note

   For RHEL 10 repositories, enable the target OS version, for example, RHEL 10.0, of each repository. If you enable only the RHEL 10 version of the repositories, the in-place upgrade is inhibited.

   For example, for the Intel architecture without an Extended Update Support (EUS) subscription, enable at minimum the following repositories:

   • Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)

     rhel-9-for-x86_64-appstream-rpms

     x86_64 <source_os_version>

   • Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)

     rhel-9-for-x86_64-baseos-rpms

     x86_64 <source_os_version>

   • Red Hat Enterprise Linux 10 for x86_64 - AppStream (RPMs)

     rhel-10-for-x86_64-appstream-rpms

     x86_64 <target_os_version>

   • Red Hat Enterprise Linux 10 for x86_64 - BaseOS (RPMs)

     rhel-10-for-x86_64-baseos-rpms

     x86_64 <target_os_version>

     Replace <source_os_version> and <target_os_version> with the source OS version and target OS version respectively, for example, 9.6 and 10.0.

     For other architectures, see RHEL 9 repositories and RHEL 10 repositories.

     For more information, see the Importing Content chapter in the Managing Content Guide for the particular version of Red Hat Satellite.

3. Attach the content host to a Content View containing the required RHEL 9 and RHEL 10 repositories.

   For more information, see the Managing Content Views chapter in the Managing Content Guide for the particular version of Red Hat Satellite.

Verification

1. Verify that the correct RHEL 9 and RHEL 10 repositories have been added to the correct Content View on Satellite Server.

   1. In the Satellite web UI, navigate to Content > Lifecycle > Content Views and click the name of the Content View.

   2. Click the Repositories tab and verify that the repositories appear as expected.

      Note

      You can also verify that the repositories have been added to the Content View using the following commands:

      Copy to Clipboard Toggle word wrap

      # hammer repository list --search 'content_label ~ rhel-9' --content-view <content_view_name> --organization <organization> --lifecycle-environment <lifecycle_environment>
      # hammer repository list --search 'content_label ~ rhel-10' --content-view <content_view_name> --organization <organization> --lifecycle-environment <lifecycle_environment>

      Replace <content_view_name> with the name of the Content View, <organization> with the organization, and <lifecycle_environement> with the name of the lifecycle environment..

2. Verify that the correct RHEL 10 repositories are enabled in the activation key associated with the Content View:

   1. In Satellite web UI navigate to Content > Lifecycle > Activation Keys and click the name of the activation key.
   2. Click the Repository Sets tab and verify that the statuses of the required repositories are Enabled.

3. Verify that all expected RHEL 9 repositories are enabled in the host. For example:

   Copy to Clipboard Toggle word wrap

   # subscription-manager repos --list-enabled | grep "^Repo ID"
   Repo ID:   rhel-9-for-x86_64-baseos-rpms
   Repo ID:   rhel-9-for-x86_64-appstream-rpms

Procedure

1. On your RHEL 9 system, perform the pre-upgrade phase:

   Copy to Clipboard Toggle word wrap

   # leapp preupgrade --target <_target_os_version_>

   Replace target_os_version with the target OS version, for example 10.0. If no target OS version is defined, Leapp uses the default target OS version specified in the table 1.1 in Supported upgrade paths.

   • If you are using custom repositories from the /etc/yum.repos.d/ directory for the upgrade, enable the selected repositories as follows:

     Copy to Clipboard Toggle word wrap

     # leapp preupgrade --enablerepo <repository_id1> --enablerepo <repository_id2> ...

     Replace repository_id with the repository IDs.

   • If you are upgrading without RHSM or by using RHUI, add the --no-rhsm option.
   • If you have an Extended Upgrade Support (EUS) or Advanced Update Support (AUS) subscription, add the --channel <channel> option. Replace <channel> with the channel name, for example, eus or aus.

   • If you are using RHEL for Real Time or the Real Time for Network Functions Virtualization (NFV) in your Red Hat OpenStack Platform, enable the deployment by using the --enablerepo option. For example:

     Copy to Clipboard Toggle word wrap

     # leapp preupgrade --enablerepo rhel-10-for-x86_64-rt-rpms

     For more information, see Configuring Real-Time Compute.

2. Examine the report in the /var/log/leapp/leapp-report.txt file and manually resolve all the reported problems. Some reported problems contain remediation suggestions. Inhibitor problems prevent you from upgrading until you have resolved them.

   The report contains the following risk factor levels:

   High

   Very likely to result in a deteriorated system state.

   Medium

   Can impact both the system and applications.

   Low

   Should not impact the system but can have an impact on applications.

   Info

   Informational with no expected impact to the system or applications.

3. In certain system configurations, the Leapp utility generates true or false questions that you must answer manually. If the pre-upgrade report contains a Missing required answers in the answer file message, complete the following steps:

   1. Open the /var/log/leapp/answerfile file and review the true or false questions.

   2. Manually edit the /var/log/leapp/answerfile file, uncomment the confirm line of the file by deleting the # symbol, and confirm your answer as True or False. For more information, see the Troubleshooting tips.

      Note

      Alternatively, you can answer the true or false question by running the following command:

      Copy to Clipboard Toggle word wrap

      # leapp answer --section <question_section>.<field_name>=<answer>

4. Repeat the previous steps to rerun the pre-upgrade report to verify that you have resolved all critical issues.

Procedure

1. Install the cockpit-leapp plug-in:

   Copy to Clipboard Toggle word wrap

   # dnf install cockpit-leapp

2. Log in to the web console as root or as a user that has permissions to enter administrative commands with sudo.

3. On your RHEL 9 system, perform the pre-upgrade phase either from the command line or from the web console terminal:

   Copy to Clipboard Toggle word wrap

   # leapp preupgrade --target <target_os_version>

   Replace target_os_version with the target OS version, for example 10.0. If no target OS version is defined, Leapp uses the default target OS version specified in the table 1.1 in Supported upgrade paths.

   • If you are using custom repositories from the /etc/yum.repos.d/ directory for the upgrade, enable the selected repositories as follows:

     Copy to Clipboard Toggle word wrap

     # leapp preupgrade --enablerepo <repository_id1> --enablerepo <repository_id2> ...

   • If you are upgrading without RHSM or by using RHUI, add the --no-rhsm option.
   • If you have an Extended Upgrade Support (EUS) or Advanced Update Support (AUS) subscription, add the --channel <channel> option. Replace <channel> with the channel name, for example, eus or`aus`.

   • If you are using RHEL for Real Time or the Real Time for Network Functions Virtualization (NFV) in your Red Hat OpenStack Platform, enable the deployment by using the --enablerepo option. For example:

     Copy to Clipboard Toggle word wrap

     # leapp preupgrade --enablerepo rhel-10-for-x86_64-rt-rpms

     For more information, see Configuring Real-Time Compute.

4. In the web console, select Upgrade Report from the navigation menu to review all reported problems. Inhibitor problems prevent you from upgrading until you have resolved them. To view a problem in detail, select the row to open the Detail pane.

   Figure 4.1. In-place upgrade report in the web console

   

   The report contains the following risk factor levels:

   High

   Very likely to result in a deteriorated system state.

   Medium

   Can impact both the system and applications.

   Low

   Should not impact the system but can have an impact on applications.

   Info

   Informational with no expected impact to the system or applications.

5. In certain configurations, the Leapp utility generates true or false questions that you must answer manually. If the Upgrade Report contains a Missing required answers in the answer file row, complete the following steps:

   1. Select the Missing required answers in the answer file row to open the Detail pane. The default answer is stated at the end of the remediation command.
   2. To confirm the default answer, select Add to Remediation Plan to execute the remediation later or Run Remediation to execute the remediation immediately.

   3. To select the non-default answer instead, execute the leapp answer command in the terminal, specifying the question you are responding to and your confirmed answer.

      Copy to Clipboard Toggle word wrap

      # leapp answer --section <question_section>.<field_name>=<answer>

      Note

      You can also manually edit the /var/log/leapp/answerfile file, uncomment the confirm line of the file by deleting the # symbol, and confirm your answer as True or False. For more information, see the Troubleshooting tips.

6. Some problems have remediation commands that you can run to automatically resolve the problems. You can run remediation commands individually or all together in the remediation command.

   1. To run a single remediation command, open the Detail pane for the problem and click Run Remediation.

   2. To add a remediation command to the remediation plan, open the Detail pane for the problem and click Add to Remediation Plan.

      Figure 4.2. Detail pane

      
   3. To run the remediation plan containing all added remediation commands, click the Remediation plan link in the top right corner above the report. Click Execute Remediation Plan to execute all listed commands.
7. After reviewing the report and resolving all reported problems, repeat steps 3-7 to rerun the report to verify that you have resolved all critical issues.

Procedure

1. Ensure that you have a full system backup or a virtual machine snapshot. You should be able to get your system to the pre-upgrade state if you follow standard disaster recovery procedures within your environment. You can use the following backup options:

   • Create a full backup of your system by using the Relax-and-Recover (ReaR) utility. For more information, see the ReaR documentation and What is Relax and Recover (ReaR) and how can I use it for disaster recovery?.

   • Create a snapshot of your system by using LVM snapshots or RAID splitting. In case of upgrading a virtual machine, you can create a snapshot of the whole VM. You can also manage snapshot and rollback boot entries by using the Boom utility. For more information, see What is BOOM and how to install it? and Managing system upgrades with snapshots.

     Note

     Because LVM snapshots do not create a full backup of your system, you might not be able to recover your system after certain upgrade failures. As a result, it is safer to create a full backup by using the ReaR utility.

2. On your RHEL 9 system, start the upgrade process:

   Copy to Clipboard Toggle word wrap

   # leapp upgrade --target <_target_os_version_>

   Replace target_os_version with the target OS version, for example 10.0. If no target OS version is defined, Leapp uses the default target OS version specified in the table 1.1 in Supported upgrade paths.

   • If you are using custom repositories from the /etc/yum.repos.d/ directory for the upgrade, enable the selected repositories as follows:

     Copy to Clipboard Toggle word wrap

     # leapp upgrade --enablerepo <repository_id1> --enablerepo <repository_id2> ...

   • If you are upgrading without RHSM or using RHUI, add the --no-rhsm option.
   • If you are upgrading by using an ISO image, add the --no-rhsm and --iso <file_path> options. Replace <file_path> with the file path to the saved ISO image, for example /home/rhel9.iso.
   • If you have an Extended Upgrade Support (EUS) or Advanced Update Support (AUS) subscription, add the --channel channel option. Replace channel with the value you used with the leapp preupgrade command, for example, eus or aus. Note that you must use the same value with the --channel option in both the leapp preupgrade and leapp upgrade commands.

   • If you are using RHEL for Real Time or the Real Time for Network Functions Virtualization (NFV) in your Red Hat OpenStack Platform, enable the deployment by using the --enablerepo option. For example:

     Copy to Clipboard Toggle word wrap

     # leapp upgrade --enablerepo rhel-10-for-x86_64-rt-rpms

     For more information, see Configuring Real-Time Compute.

3. At the beginning of the upgrade process, Leapp repeats the pre-upgrade phase described in Reviewing the pre-upgrade report.

   • If the system is upgradable, Leapp downloads necessary data and prepares an RPM transaction for the upgrade.
   • If your system does not meet the parameters for a reliable upgrade, Leapp terminates the upgrade process and provides a record describing the issue and a recommended solution in the /var/log/leapp/leapp-report.txt file. For more information, see Troubleshooting.

4. Manually reboot the system:

   Copy to Clipboard Toggle word wrap

   # reboot

   In this phase, the system boots into a RHEL 10-based initial RAM disk image, initramfs. Leapp upgrades all packages and automatically reboots to the RHEL 10 system.

   Alternatively, you can run the leapp upgrade command with the --reboot option and skip this manual step.

   If a failure occurs, investigate logs and known issues as described in Troubleshooting.

5. Log in to the RHEL 10 system and verify its state as described in Verifying the post-upgrade state.
6. Perform all post-upgrade tasks described in the upgrade report and in Performing post-upgrade tasks.

1. Remove any remaining Leapp packages from the exclude list in the /etc/dnf/dnf.conf configuration file, including the snactor package, which is a tool for upgrade extension development. During the in-place upgrade, Leapp packages that were installed with the Leapp utility are automatically added to the exclude list to prevent critical files from being removed or updated. After the in-place upgrade, these Leapp packages must be removed from the exclude list before they can be removed from the system.

   • To manually remove packages from the exclude list, edit the /etc/dnf/dnf.conf configuration file and remove the desired Leapp packages from the exclude list.

   • To remove all packages from the exclude list:

     Copy to Clipboard Toggle word wrap

     # dnf config-manager --save --setopt exclude=''

2. Remove remaining RHEL 9 packages, including remaining Leapp packages.

   1. Locate remaining RHEL 9 packages:

      Copy to Clipboard Toggle word wrap

      # rpm -qa | grep -e '\.el[789]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)' | sort

   2. Remove remaining RHEL 9 packages from your RHEL 10 system. To ensure that RPM dependencies are maintained, use DNF when performing this action. Review the transaction before accepting to ensure no packages are unintentionally removed.

      For example:

      Copy to Clipboard Toggle word wrap

      # dnf remove $(rpm -qa | grep \.el[789] | grep -vE 'gpg-pubkey|libmodulemd|katello-ca-consumer')

   3. Remove remaining Leapp dependency packages:

      Copy to Clipboard Toggle word wrap

      # dnf remove leapp-deps-el10 leapp-repository-deps-el10

3. Optional: Remove all remaining upgrade-related data from the system:

   Copy to Clipboard Toggle word wrap

   # rm -rf /var/log/leapp /root/tmp_leapp_py3 /var/lib/leapp

   Important

   Removing this data might limit Red Hat Support’s ability to investigate and troubleshoot post-upgrade problems.

4. Disable DNF repositories whose packages are not RHEL 10-compatible. Repositories managed by RHSM are handled automatically. To disable these repositories:

   Copy to Clipboard Toggle word wrap

   # dnf config-manager --set-disabled <repository_id>

   Replace repository_id with the repository ID.

5. Replace the old rescue kernel and initial RAM disk with the current kernel and disk:

   1. Remove the existing rescue kernel and initial RAM disk:

      Copy to Clipboard Toggle word wrap

      # rm /boot/vmlinuz-*rescue* /boot/initramfs-*rescue* 

   2. Reinstall the rescue kernel and related initial RAM disk:

      Copy to Clipboard Toggle word wrap

      # /usr/lib/kernel/install.d/51-dracut-rescue.install add "$(uname -r)" /boot "/boot/vmlinuz-$(uname -r)"

   3. If your system is on the IBM Z architecture, update the zipl boot loader:

      Copy to Clipboard Toggle word wrap

      # zipl

6. . Optional: Check existing configuration files:

   • Review, remediate, and then remove the rpmnew, rpmsave, and leappsave files. Note that rpmsave and leappsave are equivalent and can be handled similarly. For more information, see What are rpmnew & rpmsave files?
   • Remove configuration files for RHEL 9 DNF modules from the /etc/dnf/modules.d/ directory that are no longer valid. Note that these files have no effect on the system when related DNF modules do not exist.
7. Re-evaluate and re-apply your security policies. Especially, change the SELinux mode to enforcing. For details, see Applying security policies.

Verification

1. Verify that the previously removed rescue kernel and rescue initial RAM disk files have been created for the current kernel:

   Copy to Clipboard Toggle word wrap

   # ls /boot/vmlinuz-*rescue* /boot/initramfs-*rescue* 
   # lsinitrd /boot/initramfs-*rescue*.img | grep -qm1 "$(uname -r)/kernel/" && echo "OK" || echo "FAIL"

2. Verify the rescue boot entry refers to the existing rescue files. See the grubby output:

   Copy to Clipboard Toggle word wrap

   # grubby --info /boot/vmlinuz-*rescue*

3. Review the grubby output and verify that no RHEL 9 boot entries are configured:

   Copy to Clipboard Toggle word wrap

   # grubby --info ALL

4. Verify that no files related to previous RHEL are present in the /boot/loader/entries file:

   Copy to Clipboard Toggle word wrap

   # grep -r ".el9" "/boot/loader/entries/" || echo "Everything seems ok."

Procedure

1. Ensure that there are no SELinux denials, for example, by using the ausearch utility:

   Copy to Clipboard Toggle word wrap

   # ausearch -m AVC,USER_AVC -ts boot

   Note that the previous step covers only the most common scenario. To check for all possible SELinux denials, see the Identifying SELinux denials section in the Using SELinux title, which provides a complete procedure.

2. Open the /etc/selinux/config file in a text editor of your choice, for example:

   Copy to Clipboard Toggle word wrap

   # vi /etc/selinux/config

3. Configure the SELINUX=enforcing option:

   Copy to Clipboard Toggle word wrap

   # This file controls the state of SELinux on the system.
   # SELINUX= can take one of these three values:
   #       enforcing - SELinux security policy is enforced.
   #       permissive - SELinux prints warnings instead of enforcing.
   #       disabled - No SELinux policy is loaded.
   SELINUX=enforcing
   # SELINUXTYPE= can take one of these two values:
   #       targeted - Targeted processes are protected,
   #       mls - Multi Level Security protection.
   SELINUXTYPE=targeted

4. Save the change, and restart the system:

   Copy to Clipboard Toggle word wrap

   # reboot

Verification

1. After the system restarts, confirm that the getenforce command returns Enforcing:

   Copy to Clipboard Toggle word wrap

   $ getenforce
   Enforcing

Procedure

1. Find the appropriate security compliance data stream .xml file:

   Copy to Clipboard Toggle word wrap

   $ ls /usr/share/xml/scap/ssg/content/
   ...
   ssg-rhel10-ds.xml
   ...

   See the Viewing profiles for configuration compliance section for more information.

2. Remediate the system according to the selected profile from the appropriate data stream:

   Copy to Clipboard Toggle word wrap

   # oscap xccdf eval --profile <profile_ID> --remediate /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml

   Replace <profile_ID> with the ID of the profile according to which you want to harden your system. For a full list of profiles supported in RHEL 10, see SCAP Security Guide profiles supported in RHEL 10.

   Warning

   If not used carefully, running the system evaluation with the --remediate option enabled might render the system non-functional. Red Hat does not provide any automated method to revert changes made by security-hardening remediations. Remediations are supported on RHEL systems in the default configuration. If your system has been altered after the installation, running remediation might not make it compliant with the required security profile.

3. Restart your system:

   Copy to Clipboard Toggle word wrap

   # reboot

Verification

1. Verify that the system is compliant with the profile, and save the results in an HTML file:

   Copy to Clipboard Toggle word wrap

   $ oscap xccdf eval --report pcidss_report.html --profile pci-dss /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml

Procedure

1. Back up your *.conf files stored in the /etc/usbguard/ directory.
2. Use the usbguard generate-policy to generate a new policy file. Note that the command generates rules for the currently present USB devices only.

3. Compare the newly generated rules against the rules in the previous policy:

   1. If you identify differences in the rules for the devices that were present when you generated the new policy and the pre-upgrade rules for the same devices, modify the original rules correspondingly also for devices that might be inserted later.
   2. If there are no differences between the newly generated and the pre-upgrade rules, you can use the policy files created in RHEL 9 without any modification.

1. Stop the service:

   Copy to Clipboard Toggle word wrap

   # systemctl stop fapolicyd

2. Delete the database:

   Copy to Clipboard Toggle word wrap

   # fapolicyd-cli --delete-db

3. Start the service:

   Copy to Clipboard Toggle word wrap

   # systemctl start fapolicyd