Search
SupportConsoleDevelopersStart a trialContact

Chapter 22. Applying patches with kernel live patching

download PDF

You can use the Red Hat Enterprise Linux kernel live patching solution to patch a running kernel without rebooting or restarting any processes.

With this solution, system administrators:

  • Can immediately apply critical security patches to the kernel.
  • Do not have to wait for long-running tasks to complete, for users to log off, or for scheduled downtime.
  • Control the system’s uptime more and do not sacrifice security or stability.

Note that not every critical or important CVE will be resolved using the kernel live patching solution. Our goal is to reduce the required reboots for security-related patches, not to eliminate them entirely. For more details about the scope of live patching, see the Customer Portal Solutions article.

Warning

Some incompatibilities exist between kernel live patching and other kernel subcomponents. Read the

Limitations of kpatch carefully before using kernel live patching.

Note

For details about the support cadence of kernel live patching updates, see:

22.1. Limitations of kpatch

  • The kpatch feature is not a general-purpose kernel upgrade mechanism. It is used for applying simple security and bug fix updates when rebooting the system is not immediately possible.
  • Do not use the SystemTap or kprobe tools during or after loading a patch. The patch could fail to take effect until after such probes have been removed.

22.2. Support for third-party live patching

The kpatch utility is the only kernel live patching utility supported by Red Hat with the RPM modules provided by Red Hat repositories. Red Hat will not support any live patches which were not provided by Red Hat itself.

If you require support for an issue that arises with a third-party live patch, Red Hat recommends that you open a case with the live patching vendor at the outset of any investigation in which a root cause determination is necessary. This allows the source code to be supplied if the vendor allows, and for their support organization to provide assistance in root cause determination prior to escalating the investigation to Red Hat Support.

For any system running with third-party live patches, Red Hat reserves the right to ask for reproduction with Red Hat shipped and supported software. In the event that this is not possible, we require a similar system and workload be deployed on your test environment without live patches applied, to confirm if the same behavior is observed.

For more information about third-party software support policies, see How does Red Hat Global Support Services handle third-party software, drivers, and/or uncertified hardware/hypervisors or guest operating systems?

22.3. Access to kernel live patches

Kernel live patching capability is implemented as a kernel module (kmod) that is delivered as an RPM package.

All customers have access to kernel live patches, which are delivered through the usual channels. However, customers who do not subscribe to an extended support offering will lose access to new patches for the current minor release once the next minor release becomes available. For example, customers with standard subscriptions will only be able to live patch RHEL 8.2 kernel until the RHEL 8.3 kernel is released.

22.4. Components of kernel live patching

The components of kernel live patching are as follows:

Kernel patch module

  • The delivery mechanism for kernel live patches.
  • A kernel module which is built specifically for the kernel being patched.
  • The patch module contains the code of the desired fixes for the kernel.
  • The patch modules register with the livepatch kernel subsystem and provide information about original functions to be replaced, with corresponding pointers to the replacement functions. Kernel patch modules are delivered as RPMs.
  • The naming convention is kpatch_<kernel version>_<kpatch version>_<kpatch release>. The "kernel version" part of the name has dots replaced with underscores.
The kpatch utility
A command-line utility for managing patch modules.
The kpatch service
A systemd service required by multiuser.target. This target loads the kernel patch module at boot time.
The kpatch-dnf package
A DNF plugin delivered in the form of an RPM package. This plugin manages automatic subscription to kernel live patches.

22.5. How kernel live patching works

The kpatch kernel patching solution uses the livepatch kernel subsystem to redirect old functions to new ones. When a live kernel patch is applied to a system, the following things happen:

  1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
  2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
  3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Figure 22.1. How kernel live patching works

rhel kpatch overview

22.6. Subscribing the currently installed kernels to the live patching stream

A kernel patch module is delivered in an RPM package, specific to the version of the kernel being patched. Each RPM package will be cumulatively updated over time.

The following procedure explains how to subscribe to all future cumulative live patching updates for a given kernel. Because live patches are cumulative, you cannot select which individual patches are deployed for a given kernel.

Warning

Red Hat does not support any third party live patches applied to a Red Hat supported system.

Prerequisites

  • Root permissions

Procedure

  1. Optionally, check your kernel version: 

    # uname -r
4.18.0-94.el8.x86_64

  2. Search for a live patching package that corresponds to the version of your kernel: 

    # yum search $(uname -r)

  3. Install the live patching package: 

    # yum install "kpatch-patch = $(uname -r)"

    The command above installs and applies the latest cumulative live patches for that specific kernel only.

    If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

    The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

    Note

    An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-4_18_0-94-0-0.el8.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Verification

  • Verify that all installed kernels have been patched: 

    # kpatch list
Loaded patch modules:
kpatch_4_18_0_94_1_1 [enabled]

Installed patch modules:
kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)
…​

    The output shows that the kernel patch module has been loaded into the kernel that is now patched with the latest fixes from the kpatch-patch-4_18_0-94-1-1.el8.x86_64.rpm package.

    Note

    Entering the kpatch list command does not return an empty live patching package. Use the rpm -qa | grep kpatch command instead. 

    # rpm -qa | grep kpatch
kpatch-patch-4_18_0-477_21_1-0-0.el8_8.x86_64
kpatch-dnf-0.9.7_0.4-2.el8.noarch
kpatch-0.9.7-2.el8.noarch

Additional resources

22.7. Automatically subscribing any future kernel to the live patching stream

You can use the kpatch-dnf YUM plugin to subscribe your system to fixes delivered by the kernel patch module, also known as kernel live patches. The plugin enables automatic subscription for any kernel the system currently uses, and also for kernels to-be-installed in the future.

Prerequisites

  • You have root permissions.

Procedure

  1. Optionally, check all installed kernels and the kernel you are currently running: 

    # yum list installed | grep kernel
Updating Subscription Management repositories.
Installed Packages
...
kernel-core.x86_64         4.18.0-240.10.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
kernel-core.x86_64         4.18.0-240.15.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
...

# uname -r
4.18.0-240.10.1.el8_3.x86_64

  2. Install the kpatch-dnf plugin: 

    # yum install kpatch-dnf

  3. Enable automatic subscription to kernel live patches: 

    # yum kpatch auto
Updating Subscription Management repositories.
Last metadata expiration check: 19:10:26 ago on Wed 10 Mar 2021 04:08:06 PM CET.
Dependencies resolved.
==================================================
 Package                             Architecture
==================================================
Installing:
 kpatch-patch-4_18_0-240_10_1        x86_64
 kpatch-patch-4_18_0-240_15_1        x86_64

Transaction Summary
===================================================
Install  2 Packages
…​

    This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

    In the future, when you update the kernel, live patches will automatically be installed during the new kernel installation process.

    The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during future reboots.

    Note

    An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-4_18_0-240-0-0.el8.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Verification

  • Verify that all installed kernels have been patched: 

    # kpatch list
Loaded patch modules:
kpatch_4_18_0_240_10_1_0_1 [enabled]

Installed patch modules:
kpatch_4_18_0_240_10_1_0_1 (4.18.0-240.10.1.el8_3.x86_64)
kpatch_4_18_0_240_15_1_0_2 (4.18.0-240.15.1.el8_3.x86_64)

    The output shows that both the kernel you are running, and the other installed kernel have been patched with fixes from kpatch-patch-4_18_0-240_10_1-0-1.rpm and kpatch-patch-4_18_0-240_15_1-0-1.rpm packages respectively.

    Note

    Entering the kpatch list command does not return an empty live patching package. Use the rpm -qa | grep kpatch command instead. 

    # rpm -qa | grep kpatch
kpatch-patch-4_18_0-477_21_1-0-0.el8_8.x86_64
kpatch-dnf-0.9.7_0.4-2.el8.noarch
kpatch-0.9.7-2.el8.noarch

Additional resources

  • kpatch(1) and dnf-kpatch(8) manual pages

22.8. Disabling automatic subscription to the live patching stream

When you subscribe your system to fixes delivered by the kernel patch module, your subscription is automatic. You can disable this feature, and thus disable automatic installation of kpatch-patch packages.

Prerequisites

  • You have root permissions.

Procedure

  1. Optionally, check all installed kernels and the kernel you are currently running: 

    # yum list installed | grep kernel
Updating Subscription Management repositories.
Installed Packages
...
kernel-core.x86_64         4.18.0-240.10.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
kernel-core.x86_64         4.18.0-240.15.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
...

# uname -r
4.18.0-240.10.1.el8_3.x86_64

  2. Disable automatic subscription to kernel live patches: 

    # yum kpatch manual
Updating Subscription Management repositories.

Verification step

  • You can check for the successful outcome: 

    # yum kpatch status
...
Updating Subscription Management repositories.
Last metadata expiration check: 0:30:41 ago on Tue Jun 14 15:59:26 2022.
Kpatch update setting: manual

Additional resources

  • kpatch(1) and dnf-kpatch(8) manual pages

22.9. Updating kernel patch modules

Since kernel patch modules are delivered and applied through RPM packages, updating a cumulative kernel patch module is like updating any other RPM package.

Prerequisites

Procedure

  • Update to a new cumulative version for the current kernel: 

    # yum update "kpatch-patch = $(uname -r)"

    The command above automatically installs and applies any updates that are available for the currently running kernel. Including any future released cumulative live patches.

  • Alternatively, update all installed kernel patch modules: 

    # yum update "kpatch-patch"
Note

When the system reboots into the same kernel, the kernel is automatically live patched again by the kpatch.service systemd service.

Additional resources

22.10. Removing the live patching package

Disable the Red Hat Enterprise Linux kernel live patching solution by removing the live patching package.

Prerequisites

  • Root permissions
  • The live patching package is installed.

Procedure

  1. Select the live patching package. 

    # yum list installed | grep kpatch-patch
kpatch-patch-4_18_0-94.x86_64        1-1.el8        @@commandline
…​

    The example output above lists live patching packages that you installed.

  2. Remove the live patching package. 

    # yum remove kpatch-patch-4_18_0-94.x86_64

    When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

  3. Reboot your system.

  4. Verify that the live patching package has been removed. 

    # yum list installed | grep kpatch-patch

    The command displays no output if the package has been successfully removed.

  5. Optionally, verify that the kernel live patching solution is disabled. 

    # kpatch list
Loaded patch modules:

    The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Important

Currently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.

Additional resources

22.11. Uninstalling the kernel patch module

Prevent the Red Hat Enterprise Linux kernel live patching solution from applying a kernel patch module on subsequent boots.

Prerequisites

  • Root permissions
  • A live patching package is installed.
  • A kernel patch module is installed and loaded.

Procedure

  1. Select a kernel patch module: 

    # kpatch list
Loaded patch modules:
kpatch_4_18_0_94_1_1 [enabled]

Installed patch modules:
kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)
…​

  2. Uninstall the selected kernel patch module. 

    # kpatch uninstall kpatch_4_18_0_94_1_1
uninstalling kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)

    • Note that the uninstalled kernel patch module is still loaded: 

      # kpatch list
Loaded patch modules:
kpatch_4_18_0_94_1_1 [enabled]

Installed patch modules:
<NO_RESULT>

      When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

  3. Reboot your system.

  4. Optionally, verify that the kernel patch module has been uninstalled. 

    # kpatch list
Loaded patch modules:
…​

    The example output above shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Important

Currently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.

Additional resources

  • The kpatch(1) manual page

22.12. Disabling kpatch.service

Prevent the Red Hat Enterprise Linux kernel live patching solution from applying all kernel patch modules globally on subsequent boots.

Prerequisites

  • Root permissions
  • A live patching package is installed.
  • A kernel patch module is installed and loaded.

Procedure

  1. Verify kpatch.service is enabled. 

    # systemctl is-enabled kpatch.service
enabled

  2. Disable kpatch.service. 

    # systemctl disable kpatch.service
Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

    • Note that the applied kernel patch module is still loaded: 

      # kpatch list
Loaded patch modules:
kpatch_4_18_0_94_1_1 [enabled]

Installed patch modules:
kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)
  3. Reboot your system.

  4. Optionally, verify the status of kpatch.service. 

    # systemctl status kpatch.service
● kpatch.service - "Apply kpatch kernel patches"
   Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

    The example output testifies that kpatch.service has been disabled and is not running. Thereby, the kernel live patching solution is not active.

  5. Verify that the kernel patch module has been unloaded. 

    # kpatch list
Loaded patch modules:
<NO_RESULT>

Installed patch modules:
kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)

    The example output above shows that a kernel patch module is still installed but the kernel is not patched.

Important

Currently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.

Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.