Chapter 8. Applying patches with kernel live patching
You can use the Red Hat Enterprise Linux kernel live patching solution to patch a running kernel without rebooting or restarting any processes.
With this solution, system administrators:
- Can immediately apply critical security patches to the kernel.
- Do not have to wait for long-running tasks to complete, for users to log off, or for scheduled downtime.
- Control the system’s uptime more and do not sacrifice security or stability.
By using the kernel live patching, you can reduce the number of reboots required for security patches. However, note that you cannot address all critical or important CVEs. For more details about the scope of live patching, see the Red Hat Knowledgebase solution Is live kernel patch (kpatch) supported in Red Hat Enterprise Linux?.
Some incompatibilities exist between kernel live patching and other kernel subcomponents. Read the Limitations of kpatch carefully before using kernel live patching.
For details about the support cadence of kernel live patching updates, see:
8.1. Limitations of kpatch Copy linkLink copied to clipboard!
-
By using the
kpatchfeature, you can apply simple security and bug fix updates that do not require an immediate system reboot. -
You must not use the
SystemTaporkprobetool during or after loading a patch. The patch might not take effect until the probes are removed.
8.2. Support for third-party live patching Copy linkLink copied to clipboard!
The kpatch utility is the only kernel live patching utility supported by Red Hat with the RPM modules provided by Red Hat repositories. Red Hat does not support live patches provided by a third party.
For more information about third-party software support policies, see As a customer how does Red Hat support me when I use third party components?
8.3. Access to kernel live patches Copy linkLink copied to clipboard!
A kernel module (kmod) implements kernel live patching capability and is provided as an RPM package.
You are provided an access to kernel live patches, which are delivered through the standard channels. However, if you are not subscribed to an extended support offering, you lose access to new patches for the current minor release when the next minor release becomes available. For example, in the standard subscriptions, you are able to live patch RHEL 10.1 kernel until the RHEL 10.2 kernel is released. After the release of RHEL 10.2, live patches for RHEL 10.1 are not available.
The components of kernel live patching are as follows:
- Kernel patch module
- The delivery mechanism for kernel live patches.
- A kernel module built specifically for the kernel being patched.
- The patch module contains the code of the required fixes for the kernel.
-
Patch modules register with the
livepatchkernel subsystem and specify the original functions to replace, along with pointers to the replacement functions. Kernel patch modules are delivered as RPMs. -
The naming convention is
kpatch_<kernel version>_<kpatch version>_<kpatch release>. The "kernel version" part of the name has dots replaced with underscores.
- The
kpatchutility - A command-line utility for managing patch modules.
- The
kpatchservice -
A
systemdservice required bymultiuser.target. This target loads the kernel patch module at boot time. - The
kpatch-dnfpackage - A DNF plugin delivered in the form of an RPM package. This plugin manages automatic subscription to kernel live patches.
8.4. The process of live patching kernels Copy linkLink copied to clipboard!
The kpatch kernel patching solution uses the livepatch kernel subsystem to redirect outdated functions to updated ones. Applying a live kernel patch to a system triggers the following processes:
-
The kernel patch module is copied to the
/var/lib/kpatch/directory and registered for re-application to the kernel bysystemdon next boot. -
The
kpatchmodule loads into the running kernel and the new functions are registered to theftracemechanism with a pointer to the location in memory of the new code.
When the kernel accesses the patched function, the ftrace mechanism redirects it, bypassing the original functions and leading the kernel to the patched version of the function.
Figure 8.1. How kernel live patching works
8.5. Subscribing the currently installed kernels to the live patching stream Copy linkLink copied to clipboard!
A kernel patch module is delivered in an RPM package, specific to the version of the kernel being patched. Each RPM package will be cumulatively updated over time.
The following procedure explains how to subscribe to all future cumulative live patching updates for a given kernel. Because live patches are cumulative, you cannot select which individual patches are deployed for a given kernel.
Red Hat does not support any third party live patches applied to a Red Hat supported system.
Prerequisites
- You have root permissions on the system.
Procedure
Optional: Check your kernel version:
uname -r
# uname -r 6.12.0-55.9.1.el10_0.x86_64Copy to Clipboard Copied! Toggle word wrap Toggle overflow Search for a live patching package that corresponds to the version of your kernel:
dnf search $(uname -r)
# dnf search $(uname -r)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Install the live patching package:
dnf install kpatch
# dnf install kpatchCopy to Clipboard Copied! Toggle word wrap Toggle overflow This command installs and applies the latest cumulative live patches for that specific kernel only.
If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.
The kernel patch module is also installed into the
/var/lib/kpatch/directory to be loaded by thesystemdsystem and service manager during the future reboots.NoteAn empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example
kpatch-patch-6_12_0-1-0-0.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.
Verification
Verify that all installed kernels have been patched:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The output shows that the kernel patch module has been loaded into the kernel that is now patched with the latest fixes from the
kpatch-patch-6_12_0-0.el10_0.x86_64.rpmpackage.See the
kpatch(1)man page on your system for more information.NoteEntering the
kpatch listcommand does not return an empty live patching package. Use therpm -qa | grep kpatchcommand instead.rpm -qa | grep kpatch
# rpm -qa | grep kpatch kpatch-dnf-0.4-3.el10.noarch kpatch-0.9.7-2.el10.noarch kpatch-patch-6_12_0-0.el10_0.x86_64Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.6. Automatically subscribing any future kernel to the live patching stream Copy linkLink copied to clipboard!
You can use the kpatch-dnf DNF plugin to subscribe your system to fixes delivered by the kernel patch module, also known as kernel live patches. The plugin enables automatic subscription for any kernel the system currently uses, and also for kernels to-be-installed in the future.
Prerequisites
- You have root permissions on the system.
Procedure
Optional: Check all installed kernels and the kernel you are currently running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
kpatch-dnfplugin:dnf install kpatch-dnf
# dnf install kpatch-dnfCopy to Clipboard Copied! Toggle word wrap Toggle overflow Enable automatic subscription to kernel live patches:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.
When you update the kernel, live patches are installed automatically during the new kernel installation process.
The kernel patch module is also installed into the
/var/lib/kpatch/directory that is loaded by thesystemdsystem and service manager during future reboots.NoteAn empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example
kpatch-patch-6_12_0-1-0-0.el10.x86_64.rpm.The installation of the empty RPM subscribes the system to all future live patches for the given kernel.
Verification
Verify that all installed kernels are patched:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The output shows that both the kernel you are running, and the other installed kernel have been patched with fixes from
kpatch-patch-6_12_0-1-0-1.el10.x86_64.rpmandkpatch-patch-6_12_0-2-0-1.el10.x86_64.rpmpackages.NoteEntering the
kpatch listcommand does not return an empty live patching package. Use therpm -qa | grep kpatchcommand instead.rpm -qa | grep kpatch
# rpm -qa | grep kpatch kpatch-dnf-0.9.7_0.4-4.el10.noarch kpatch-0.9.7-4.el10.noarch kpatch-patch-6_12_0_1-0-0.el10_0.x86_64Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.7. Disabling automatic subscription to the live patching stream Copy linkLink copied to clipboard!
When you subscribe your system to fixes delivered by the kernel patch module, your subscription is automatic. You can disable this feature, to disable automatic installation of kpatch-patch packages.
Prerequisites
- You have root permissions on the system.
Procedure
Optional: Check all installed kernels and the kernel you are currently running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Disable automatic subscription to kernel live patches:
dnf kpatch manual
# dnf kpatch manual Updating Subscription Management repositories.Copy to Clipboard Copied! Toggle word wrap Toggle overflow See
kpatch(1)anddnf-kpatch(8)manual pages for more information.
Verification
You can check for the successful outcome:
yum kpatch status
# yum kpatch status ... Updating Subscription Management repositories. Last metadata expiration check: 0:30:41 ago on Tue Jun 14 15:59:26 2022. Kpatch update setting: manualCopy to Clipboard Copied! Toggle word wrap Toggle overflow
8.8. Updating kernel patch modules Copy linkLink copied to clipboard!
The kernel patch modules are delivered and applied through RPM packages. The process of updating a cumulative kernel patch module is similar to updating any other RPM package.
Prerequisites
- The system is subscribed to the live patching stream, as described in Subscribing the currently installed kernels to the live patching stream.
Procedure
Update to a new cumulative version for the current kernel:
dnf update kpatch
# dnf update kpatchCopy to Clipboard Copied! Toggle word wrap Toggle overflow This command automatically installs and applies any updates that are available for the currently running kernel. Including any future released cumulative live patches.
Alternatively, update all installed kernel patch modules:
dnf update kpatch
# dnf update kpatchCopy to Clipboard Copied! Toggle word wrap Toggle overflow
When the system reboots into the same kernel, the kernel is automatically live patched again by the kpatch.service systemd service.
8.9. Removing the live patching package Copy linkLink copied to clipboard!
Disable the Red Hat Enterprise Linux kernel live patching solution by removing the live patching package.
Prerequisites
- You have root permissions on the system.
- The live patching package is installed.
Procedure
Select the live patching package:
dnf list installed | grep kpatch-patch
# dnf list installed | grep kpatch-patch kpatch-patch-6.12.0-0.el10_0.x86_64 0-0.el10 @@commandline …Copy to Clipboard Copied! Toggle word wrap Toggle overflow The example output lists live patching packages that you installed.
Remove the live patching package:
dnf remove kpatch-patch-6.12.0-0.el10_0.x86_64
# dnf remove kpatch-patch-6.12.0-0.el10_0.x86_64Copy to Clipboard Copied! Toggle word wrap Toggle overflow When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.
- Reboot your system.
Verify the live patching package is removed:
dnf list installed | grep kpatch-patch
# dnf list installed | grep kpatch-patchCopy to Clipboard Copied! Toggle word wrap Toggle overflow The command displays no output if the package has been successfully removed.
Verification
Verify the kernel live patching solution is disabled:
kpatch list
# kpatch list Loaded patch modules:Copy to Clipboard Copied! Toggle word wrap Toggle overflow The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.
Currently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.
8.10. Uninstalling the kernel patch module Copy linkLink copied to clipboard!
Prevent the Red Hat Enterprise Linux kernel live patching solution from applying a kernel patch module on subsequent boots.
Prerequisites
- You have root permissions on the system.
- A live patching package is installed.
- A kernel patch module is installed and loaded.
Procedure
Select a kernel patch module:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Uninstall the selected kernel patch module.
kpatch uninstall kpatch_6_12_0_1_0_1
# kpatch uninstall kpatch_6_12_0_1_0_1 uninstalling kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note that the uninstalled kernel patch module is still loaded:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.
- Reboot your system.
Verification
Verify that the kernel patch module is uninstalled:
kpatch list
# kpatch list Loaded patch modules: …Copy to Clipboard Copied! Toggle word wrap Toggle overflow This example output shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.
8.11. Disabling kpatch.service Copy linkLink copied to clipboard!
Prevent the Red Hat Enterprise Linux kernel live patching solution from applying all kernel patch modules globally on subsequent boots.
Prerequisites
- You have root permissions on the system.
- A live patching package is installed.
- A kernel patch module is installed and loaded.
Procedure
Verify
kpatch.serviceis enabled.systemctl is-enabled kpatch.service
# systemctl is-enabled kpatch.service enabledCopy to Clipboard Copied! Toggle word wrap Toggle overflow Disable
kpatch.service:systemctl disable kpatch.service
# systemctl disable kpatch.service Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note that the applied kernel patch module is still loaded:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- Reboot your system.
Optional: Verify the status of
kpatch.service.systemctl status kpatch.service
# systemctl status kpatch.service ● kpatch.service - "Apply kpatch kernel patches" Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled) Active: inactive (dead)Copy to Clipboard Copied! Toggle word wrap Toggle overflow The example output testifies that
kpatch.serviceis disabled. Thereby, the kernel live patching solution is not active.Verify that the kernel patch module has been unloaded.
kpatch list
# kpatch list Loaded patch modules: Installed patch modules: kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)Copy to Clipboard Copied! Toggle word wrap Toggle overflow The example output shows that a kernel patch module is still installed but the kernel is not patched.
ImportantCurrently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.