Recover a system with rescue mode

The installation program’s rescue mode is a minimal Linux environment that you can boot from the Red Hat Enterprise Linux DVD or other boot media. It contains command-line utilities for repairing a wide variety of issues.

Rescue mode can be accessed from the Troubleshooting menu of the boot menu. In this mode, you can mount file systems as read-only, denylist or add a driver provided on a driver disc, install or upgrade system packages, or manage partitions.

Note

The installation program’s rescue mode is different from rescue mode (an equivalent to single-user mode) and emergency mode, which are provided as parts of the systemd system and service manager.

To boot into rescue mode, you must boot the system by using Red Hat Enterprise Linux boot media, such as a minimal boot disc, USB drive, or full installation DVD.

Important

Configure advanced storage, such as iSCSI or zFCP devices, by using dracut boot options such as rd.zfcp= or root=iscsi:options. Alternatively, use the CMS configuration file on 64-bit IBM Z. You cannot configure these storage devices interactively after booting into rescue mode. For information about dracut boot options, see the dracut.cmdline(7) man page on your system.

Boot into rescue mode
Copy link

Boot your RHEL system into rescue mode to troubleshoot and repair system issues when the normal boot process fails. Rescue mode provides a minimal environment to diagnose problems, recover data, and fix configuration issues that prevent normal system operation.

Procedure
Copy link

  1. Boot the system from either minimal boot media, or a full installation DVD or USB drive, and wait for the boot menu to be displayed.
  2. From the boot menu, either select Troubleshooting > Rescue a Red Hat Enterprise Linux system option, or append the inst.rescue option to the boot command line. To enter the boot command line, press the Tab key on BIOS-based systems or the e key on UEFI-based systems.
  3. Optional: If your system requires a third-party driver provided on a driver disc to boot, append the inst.dd=driver_name to the boot command line: 
    inst.rescue inst.dd=driver_name
  4. Optional: If a driver that is part of the Red Hat Enterprise Linux distribution prevents the system from booting, append the modprobe.blacklist= option to the boot command line: 
    inst.rescue modprobe.blacklist=driver_name
  5. Press Enter (BIOS-based systems) or Ctrl+X (UEFI-based systems) to boot the modified option. Wait until the following message is displayed: 
    The rescue environment now attempts to find your Linux installation and mount it under the directory: /mnt/sysroot/. You can then make any changes required to your system. Choose 1 to proceed with this step. You can choose to mount your file systems read-only instead of read-write by choosing 2. If for some reason this process does not work, choose 3 to skip directly to a shell.

1) Continue
2) Read-only mount
3) Skip to shell
4) Quit (Reboot)

    If you select 1, the installation program attempts to mount your file system under the directory /mnt/sysroot/. You are notified if it fails to mount a partition. If you select 2, it attempts to mount your file system under the directory /mnt/sysroot/, but in read-only mode. If you select 3, your file system is not mounted.

    For the system root, the installer supports two mount points /mnt/sysimage and /mnt/sysroot. The /mnt/sysroot path is used to mount / of the target system. Usually, the physical root and the system root are the same, so /mnt/sysroot is attached to the same file system as /mnt/sysimage. The only exceptions are rpm-ostree systems, where the system root changes based on the deployment. Then, /mnt/sysroot is attached to a subdirectory of /mnt/sysimage. Use /mnt/sysroot for chroot.

  6. Select 1 to continue. Once your system is in rescue mode, a prompt appears on VC (virtual console) 1 and VC 2. Use the Ctrl+Alt+F1 key combination to access VC 1 and Ctrl+Alt+F2 to access VC 2: 
    sh-4.2#
  7. Even if your file system is mounted, the default root partition in rescue mode is a temporary root partition. It is not the root partition used during normal user mode (multi-user.target or graphical.target). If you selected to mount your file system and it mounted successfully, change the rescue mode root partition to your file system root partition. Run the following command: 
    sh-4.2# chroot /mnt/sysroot

    This is useful if you need to run commands, such as rpm, that require your root partition to be mounted as /. To exit the chroot environment, type exit to return to the prompt.

  8. If you selected 3, you can still try to mount a partition or LVM2 logical volume manually inside rescue mode. To do so, create a directory, such as /directory/, and run the following command: 
    sh-4.2# mount -t xfs /dev/mapper/VolGroup00-LogVol02 /directory

    In this command, /directory/ is the directory that you created and /dev/mapper/VolGroup00-LogVol02 is the LVM2 logical volume you want to mount. If the partition is a different type than XFS, replace the xfs string with the correct type (such as ext4).

  9. If you do not know the names of all physical partitions, use the following command to list them: 
    sh-4.2# fdisk -l

    If you do not know the names of all LVM2 physical volumes, volume groups, or logical volumes, use the pvdisplay, vgdisplay or lvdisplay commands.

Generate an SOS report in rescue mode
Copy link

The sos report command-line utility collects configuration and diagnostic data, including the kernel version, loaded modules, and system and service configuration files. The output is saved as a tar archive in /var/tmp/. It helps analyze, troubleshoot errors, and can be captured in rescue mode.

Before you begin
Copy link

  • You have booted into rescue mode.
  • You have mounted the installed system / (root) partition in read-write mode.
  • You have contacted Red Hat Support about your case and received a case number.

Procedure
Copy link

  1. Change the root directory to the /mnt/sysroot/ directory: 
    sh-4.2# chroot /mnt/sysroot/
  2. Run the sos report command to generate an archive with system configuration and diagnostic information: 
    sh-4.2# sos report

    sos report prompts you to enter your name and the case number you received from Red Hat Support. Use only letters and numbers because adding any of the following characters or spaces could render the report unusable: # % & { } \ < > > * ? / $ ~ ' " : @ + ` | =

  3. Optional: If you want to transfer the generated archive to a new location by using the network, it is necessary to have a network interface configured. In this scenario, use the dynamic IP addressing as no other steps are required. However, when using static addressing, enter the following command to assign an IP address (for example 10.13.153.64/23) to a network interface, for example dev eth0: 
    bash-4.2# ip addr add 10.13.153.64/23 dev eth0
  4. Exit the chroot environment: 
    sh-4.2# exit
  5. Store the generated archive in a new location, from where it can be easily accessible: 
    sh-4.2# cp /mnt/sysroot/var/tmp/sos report new_location
  6. For transferring the archive through the network, use the scp utility: 
    sh-4.2# scp /mnt/sysroot/var/tmp/sos report username@hostname:sos report

    Additional resources

Reinstall the GRUB2 boot loader
Copy link

In some scenarios, the GRUB2 boot loader is mistakenly deleted, corrupted, or replaced by other operating systems. You can reinstall GRUB2 on the master boot record (MBR) on AMD64 and Intel 64 systems with BIOS.

Before you begin
Copy link

  • You have booted into rescue mode.
  • You have mounted the installed system / (root) partition in read-write mode.
  • You have mounted the /boot mount point in read-write mode.

Procedure
Copy link

  1. Change the root partition: 
    sh-4.2# chroot /mnt/sysroot/
  2. Reinstall the GRUB2 boot loader, where the install_device block device was installed: 
    sh-4.2# /sbin/grub2-install install_device
    Important

    Running the grub2-install command could lead to the machine being unbootable if all the following conditions apply:

    • The system is an AMD64 or Intel 64 with Extensible Firmware Interface (EFI).
    • Secure Boot is enabled.

    After you run the grub2-install command, you cannot boot the AMD64 or Intel 64 systems that have Extensible Firmware Interface (EFI) and Secure Boot enabled. This issue occurs because the grub2-install command installs an unsigned GRUB2 image that boots directly instead of using the shim application. When the system boots, the shim application validates the image signature, which when not found fails to boot the system.
  3. Reboot the system.

Add or remove a driver by using dnf
Copy link

You can use the dnf package manager in rescue mode to add updated or missing drivers, or remove malfunctioning drivers that prevent the system from booting.

Before you begin
Copy link

  • You have booted into rescue mode.
  • You have mounted the installed system in read-write mode.

Procedure
Copy link

  1. To add a new driver using dnf:
    1. Make the RPM package that contains the driver available. For example, mount a CD or USB flash drive and copy the RPM package to a location of your choice under /mnt/sysroot/, for example: /mnt/sysroot/root/drivers/.
    2. Change the root directory to /mnt/sysroot/: 
      sh-4.2# chroot /mnt/sysroot/
    3. Use the dnf install command to install the driver package. For example, run the following command to install the xorg-x11-drv-wacom driver package from /root/drivers/: 
      sh-4.2# dnf install /root/drivers/xorg-x11-drv-wacom-0.23.0-6.el7.x86_64.rpm
      Note

      The /root/drivers/ directory in this chroot environment is the /mnt/sysroot/root/drivers/ directory in the original rescue environment.
    4. Exit the chroot environment: 
      sh-4.2# exit
  2. To remove a driver by using dnf:
    1. Change the root directory to the /mnt/sysroot/ directory: 
      sh-4.2# chroot /mnt/sysroot/
    2. Use the dnf remove command to remove the driver package. For example, to remove the xorg-x11-drv-wacom driver package, run: 
      sh-4.2# dnf remove xorg-x11-drv-wacom
    3. Exit the chroot environment: 
      sh-4.2# exit

      If you cannot remove a malfunctioning driver for some reason, you can instead blocklist the driver so that it does not load at boot time.

    4. After you remove the drivers, reboot the system.
      Warning

      When you install a driver from a driver disk, the driver disk updates all initramfs images on the system to use this driver. If a problem with a driver prevents a system from booting, you cannot rely on booting the system from another initramfs image.