Red Hat Summit14장. Performing soft reboots to RHEL bootc images
You can restart a device from within the operating system, without re-initializing the boot loader and kernel, by using a systemd soft-reboot. Unlike a hard reboot, it only cycles the userspace. This enables for fast cycling of services and applications without the time-consuming task of a full hard reboot.
A soft reboot primarily resolves common software-related problems with minimal risk. It is a standard troubleshooting step for issues such as:
- Minor software glitches: A soft reboot resolves temporary errors that cause applications to freeze, crash, or misbehave.
- Performance issues: Over time, open applications and background processes consume resources and slow down a system. A soft reboot clears these processes and refreshes the RAM, which can improve performance.
- Network problems: For issues such as connectivity drops, a restart refreshes the network settings of the system.
- Completing updates: Some system updates require a reboot to be finalized properly.
In a RHEL environment with systemd, a soft reboot restarts the userspace while leaving the kernel and hardware running, which offers a significant benefits over a traditional, full reboot:
-
Reduced downtime: A soft reboot significantly reduces downtime because it skips the time-consuming boot process involving the BIOS/UEFI, boot loader, kernel, and initial ramdisk (
initrd) . The system can quickly become responsive again, which is important for critical servers where minimizing downtime is essential. - Improved operational efficiency: For software and configuration changes that do not involve the kernel, you can use soft reboot to apply updates without a full system restart. This is particularly useful for containerized or image-based systems where a new root filesystem snapshot can activate instantly.
When invoked, the systemd-soft-reboot.service system service performs the following actions:
-
Sends a
SIGTERMsignal to any processes left running, without waiting for the processes to exit. -
Continues with a
SIGKILLsignal to terminate any process immediately. -
If the
/run/nextroot/directory exists, which can be a regular directory, a directory mount point, or a symlink to either, it switches the file system root to it. - Re-runs the service manager from the root file system, which might have been updated, and enqueues a new boot transaction as in a normal reboot.
-
Soft-reboot.targetpulls in this service and performs a userspace-only reboot operation.
14.1. Soft reboot behavior in package and image mode
A system running in image mode performs a soft reboot in the same way as a package mode system that reboots only userspace. The difference is that you must build the updates to these packages and services first on the container image. The examples demonstrate how the packages and service updates are applied and restarted.
Soft reboot behavior in the package mode
In a package-based system, for example, a RHEL system that uses dnf, the soft reboot process involves shutting down and restarting systemd units to load new libraries and binaries. You can apply updates with minimal downtime. When you apply updates by using dnf in RHEL, systemd manages the behavior of the system during a soft reboot. The key units and expected behaviors:
-
Service restarts: A soft reboot triggers
systemdto terminate and restart all running services. For updated packages that provide a service, for example, a web server or database, the service reloads the new binaries upon restarting, effectively applying the patch. -
Unit dependencies:
systemdshuts down and starts units based on their defined dependencies and ordering. A soft reboot ensures these relationships are maintained, minimizing the chance of an improper shutdown sequence. -
Processes and libraries: For updated shared libraries such as
glibcoropenssl, a running process continues to use the older, mapped library until it restarts. A soft reboot ensures that all processes terminate and subsequently restart, causing them to link against the new version of the library. - Minimal downtime: A soft reboot is significantly faster than a hard reboot, because it does not reinitialize the kernel and hardware. It is effective for applying most userspace updates with minimal service interruption.
-
Command-line tools: The
dnf-plugins-corepackage includes theneeds-restartingtool. After running adnf update, you can rundnf needs-restartingto check if a soft reboot or specific service restart is required to apply the changes.
Userspace patching addresses security vulnerabilities and bugs in user-facing applications and shared libraries. When you apply patches, a soft reboot or process restart is required to load the new code. For example:
-
OpenSSL: Use case: Critical OpenSSL vulnerability discovered.
Problem: Applications that use OpenSSL, such as web servers, databases, and SSH daemons, remain exposed without a restart, continuing to use the vulnerable shared library.
Soft reboot solution: After performing a dnf update openssl, a soft reboot terminates dependent processes. Systemd then restarts these services, automatically loading the new, patched libssl.so and libcrypto.so libraries, securing the system without a full machine reboot.
-
Glibc: Use case: A bug or security flaw is found in theGNU CLibrary (glibc).
Problem: Glibc is a foundational userspace component that nearly every program on the system depends on. A vulnerability in glibc impacts the entire system. Simply restarting one or two services is not enough, as many other processes will still be vulnerable.
Soft reboot solution: A dnf update glibc followed by a soft reboot is the most reliable way to ensure that all processes restart and re-link to the new glibc. This avoids the longer downtime of a full system reboot while guaranteeing the update is applied everywhere.
-
dbus-broker:
Use case: Updating the dbus-broker daemon for security or performance.
Problem: Dbus-broker is a critical system service. While updates are usually resilient, the sensitivity of the protocol requires restarting the broker and related services.
Soft reboot solution: A soft reboot properly restarts dbus-broker and all dependent services and applications, with systemd ensuring a clean shutdown and re-initialization.
Soft reboot behaviour in Image mode
In RHEL image mode (bootc), systemd performs a userspace-only restart for a soft reboot, which is a significantly faster process than a full hardware reboot. Systemd-soft-reboot.service makes the soft reboot process in bootc images mode possible as it orchestrates a sequence that resets the userspace while leaving the kernel and underlying hardware running.
In an image-based system, you can manage updates in two ways:
- Staged updates: The system fetches these updates from a container registry and installs them to an alternate, inactive partition or filesystem. It continues to run on the old version until it initiates a reboot. When a soft reboot occurs, the system switches to the newly staged, updated root filesystem. The transition is atomic, meaning the entire userspace is replaced at once. The system boots into the new, updated OS image, and if issues appear, it retains the ability to roll back to the previous image on a subsequent boot.
- Unstaged updates: Dynamic, in-place updates to the running userspace, such as configuration changes or restarting a single service. They do not involve creating and booting from a new, complete OS image. A soft reboot reloads the current userspace from the same root file system. It does not pull or switch to a new, updated OS image. You can use it to reset the current software state without touching the kernel, which is useful for applying non-image-level changes or resolving userspace issues.
The systemd soft reboot mechanism is different from a kexec reboot. The kexec and kernel switching features are not available in the systemd soft reboot, as it only restarts the userspace, leaving the kernel untouched. This ensures continuity and avoids the potential complexities and inconsistencies that can arise from changing kernels without a hardware reset. An updated OS image that includes a new kernel version requires a traditional, full reboot.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}