Managing, monitoring, and updating the kernel

Procedure

1. List the available versions of a package:

   $ dnf repoquery <package_name>

2. Display the list of files in a package:

   $ dnf repoquery -l <package_name>

Procedure

1. To update the kernel, enter the following command:

   # dnf upgrade kernel

   This command updates the kernel along with all dependencies to the latest available version.

2. Reboot your system for the changes to take effect.

   See the dnf(8) man page on your system for more information.

Procedure

1. Identify the system architecture:

   # uname -r

   6.12.0-55.9.1.el10_0.x86_64

   In this output, x86_64 indicates a 64-bit Intel or AMD architecture.

2. Check the default page size:

   # getconf PAGE_SIZE

   4096

   On x86_64 systems, the output is 4096 B, which means the default page size is 4 KB.

   On ppc64le systems, the output is 65536 B, which means the default page size is 64 KB.

Procedure

1. Select a kernel module you want to load.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Load the relevant kernel module:

   # modprobe <pass:quotes[MODULE_NAME]>

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. List all the loaded kernel modules:

   # lsmod

2. Select the kernel module to unload.

   If a kernel module has dependencies, unload those before unloading the kernel module. For details on identifying modules with dependencies, see Listing currently loaded kernel modules and Kernel module dependencies.

3. Unload the relevant kernel module:

   # modprobe -r <pass:quotes[MODULE_NAME]>

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. Boot the system into the boot loader.
2. Use the cursor keys to highlight the relevant boot loader entry.
3. Press the e key to edit the entry.
4. Use the cursor keys to navigate to the line that starts with linux.

5. Append modprobe.blacklist=module_name to the end of the line.

   The serio_raw kernel module illustrates a rogue module to be unloaded early in the boot process.

6. Press Ctrl+X to boot using the modified configuration.

Procedure

1. Select a kernel module you want to load during the boot process.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for the module:

   # echo <MODULE_NAME> > /etc/modules-load.d/<MODULE_NAME>.conf

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Verification

1. After reboot, verify the relevant module is loaded:

   $ lsmod | grep <MODULE_NAME>

   Important

   The changes described in this procedure will persist after rebooting the system.

   See the modules-load.d(5) man page on your system for more information.

Procedure

1. List modules loaded to the currently running kernel by using the lsmod command:

   $ lsmod

   Module                  Size  Used by
   tls                   131072  0
   uinput                 20480  1
   snd_seq_dummy          16384  0
   snd_hrtimer            16384  1
   …

   In the output, identify the module you want to prevent from getting loaded.

   • Alternatively, identify an unloaded kernel module you want to prevent from potentially loading in the /lib/modules/<KERNEL-VERSION>/kernel/<SUBSYSTEM>/ directory, for example:

     $ ls /lib/modules/6.12.0-55.9.1.el10_0.x86_64/kernel/crypto/

     ansi_cprng.ko.xz        chacha20poly1305.ko.xz  md4.ko.xz               serpent_generic.ko.xz
     anubis.ko.xz            cmac.ko.xz…

2. Create a configuration file serving as a denylist:

   # touch /etc/modprobe.d/denylist.conf

3. In a text editor of your choice, combine the names of modules you want to exclude from automatic loading to the kernel with the blacklist configuration command, for example:

   # Prevents <KERNEL-MODULE-1> from being loaded
   blacklist <MODULE-NAME-1>
   install <MODULE-NAME-1> /bin/false
   
   # Prevents <KERNEL-MODULE-2> from being loaded
   blacklist <MODULE-NAME-2>
   install <MODULE-NAME-2> /bin/false
   …

   Because the blacklist command does not prevent the module from getting loaded as a dependency for another kernel module that is not in a denylist, you must also define the install line. In this case, the system runs /bin/false instead of installing the module. The lines starting with a hash sign are comments you can use to make the file more readable.

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

4. Create a backup copy of the current initial RAM disk image before rebuilding:

   # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

   • Alternatively, create a backup copy of an initial RAM disk image which corresponds to the kernel version for which you want to prevent kernel modules from automatic loading:

     # cp /boot/initramfs-<VERSION>.img /boot/initramfs-<VERSION>.img.bak.$(date +%m-%d-%H%M%S)

5. Generate a new initial RAM disk image to apply the changes:

   # dracut -f -v

   • If you build an initial RAM disk image for a different kernel version than your system currently uses, specify both target initramfs and kernel version:

     # dracut -f -v /boot/initramfs-<TARGET-VERSION>.img <CORRESPONDING-TARGET-KERNEL-VERSION>

6. Restart the system:

   $ reboot

   Important

   The changes described in this procedure will take effect and persist after rebooting the system. If you incorrectly list a key kernel module in the denylist, you can switch the system to an unstable or non-operational state.

Procedure

1. Create the /root/testmodule/test.c file with the following content.

   #include <linux/module.h>
   #include <linux/kernel.h>
   
   int init_module(void)
       { printk("Hello World\n This is a test\n"); return 0; }
   
   void cleanup_module(void)
       { printk("Good Bye World"); }
   
   MODULE_LICENSE("GPL");

   The test.c file is a source file that provides the main functionality to the kernel module. The file has been created in a dedicated /root/testmodule/ directory for organizational purposes. After the module compilation, the /root/testmodule/ directory will contain multiple files.

   The test.c file includes from the system libraries:

   • The linux/kernel.h header file is necessary for the printk() function in the example code.
   • The linux/module.h file contains function declarations and macro definitions that are shared across several source files written in C programming language.
2. Follow the init_module() and cleanup_module() functions to start and end the kernel logging function printk(), which prints text.

3. Create the /root/testmodule/Makefile file with the following content.

   obj-m := test.o

   The Makefile contains instructions for the compiler to produce an object file named test.o. The obj-m directive specifies that the resulting test.ko file is going to be compiled as a loadable kernel module. Alternatively, the obj-y directive can instruct to build test.ko as a built-in kernel module.

4. Compile the kernel module:

   # make -C /lib/modules/$(uname -r)/build M=/root/testmodule modules

   make: Entering directory '/usr/src/kernels/6.12.0-55.9.1.el10_0.x86_64'
     CC [M]  /root/testmodule/test.o
     MODPOST /root/testmodule/Module.symvers
     CC [M]  /root/testmodule/test.mod.o
     LD [M]  /root/testmodule/test.ko
     BTF [M] /root/testmodule/test.ko
   Skipping BTF generation for /root/testmodule/test.ko due to unavailability of vmlinux
   make: Leaving directory '/usr/src/kernels/6.12.0-55.9.1.el10_0.x86_64'

   The compiler creates an object file (test.o) for each source file (test.c) as an intermediate step before linking them together into the final kernel module (test.ko).

   After a successful compilation, /root/testmodule/ contains additional files that relate to the compiled custom kernel module. The compiled module itself is represented by the test.ko file.

Verification

1. Optional: check the contents of the /root/testmodule/ directory:

   # ls -l /root/testmodule/

   total 152
   -rw-r--r--. 1 root root    16 Jul 26 08:19 Makefile
   -rw-r--r--. 1 root root    25 Jul 26 08:20 modules.order
   -rw-r--r--. 1 root root     0 Jul 26 08:20 Module.symvers
   -rw-r--r--. 1 root root   224 Jul 26 08:18 test.c
   -rw-r--r--. 1 root root 62176 Jul 26 08:20 test.ko
   -rw-r--r--. 1 root root    25 Jul 26 08:20 test.mod
   -rw-r--r--. 1 root root   849 Jul 26 08:20 test.mod.c
   -rw-r--r--. 1 root root 50936 Jul 26 08:20 test.mod.o
   -rw-r--r--. 1 root root 12912 Jul 26 08:20 test.o

2. Copy the kernel module to the /lib/modules/$(uname -r)/ directory:

   # cp /root/testmodule/test.ko /lib/modules/$(uname -r)/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v test

   insmod /lib/modules/6.12.0-55.9.1.el10_0.x86_64/test.ko

5. Verify that the kernel module was successfully loaded:

   # lsmod | grep test

   test                   16384  0

6. Read the latest messages from the kernel ring buffer:

   # dmesg

   [74422.545004] Hello World
                   This is a test

Procedure

1. Boot into the GRUB boot menu.
2. Select the kernel you want to start.
3. Press the e key to edit the kernel parameters.
4. Find the kernel command line by moving the cursor down.
5. Move the cursor to the end of the line.

6. Edit the kernel parameters as required. For example, to run the system in emergency mode, add the emergency parameter at the end of the linux line:

   linux   ($root)/vmlinuz-6.12.0-0.el10_0.x86_64 root=/dev/mapper/rhel-root ro crashkernel=2G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet emergency

   To enable the system messages, remove the rhgb and quiet parameters.

7. Press Ctrl+x to boot with the selected kernel and the modified command line parameters.

   Important

   If you press the Esc key to leave command line editing, it will drop all the user made changes.

Procedure

1. Add the following two lines to the /etc/default/grub file:

   GRUB_TERMINAL="serial"
   GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

   The first line disables the graphical terminal. The GRUB_TERMINAL key overrides values of GRUB_TERMINAL_INPUT and GRUB_TERMINAL_OUTPUT keys.

   The second line adjusts the baud rate (--speed), parity and other values to fit your environment and hardware. Note that a higher baud rate, for example 115200, is preferable for tasks such as following log files.

2. Update the GRUB configuration file:

   # grub2-mkconfig -o /boot/grub2/grub.cfg

   This applies to both, BIOS and UEFI based machines.

3. Reboot the system for the changes to take effect.

Procedure

1. Add or remove a kernel parameter for individual kernels in a post installation script with grubby:

   # grubby --update-kernel <PATH_TO_KERNEL> --args "<NEW_ARGUMENTS>"

   For example, add the noapic parameter to the chosen kernel:

   # grubby --update-kernel /boot/vmlinuz-6.12.0-0.el10_0.x86_64 --args "noapic"

   The parameter is propagated into the BLS snippets, but not into the /etc/default/grub file.

2. Overwrite BLS snippets with the contents of the GRUB_CMDLINE_LINUX values present in the /etc/default/grub file:

   # grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

   Generating grub configuration file ...
   Adding boot menu entry for UEFI Firmware Settings ...
   done

   Note

   Other changes, such as changes made to GRUB_TIMEOUT key (also included in the /etc/default/grub GRUB configuration file) are propagated to the new grub.cfg file by executing grub2-mkconfig command.

Verification

1. Reboot your system.

2. Verify that the parameters are included in the /proc/cmdline file.

   For example, if you added the noapic:

   BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.12.0-0.el10_0.x86_64 root=/dev/mapper/RHELCSB-Root ro vconsole.keymap=us crashkernel=2G-64G:256M,64G-:512M rd.lvm.lv=RHELCSB/Root rd.luks.uuid=luks-d8a28c4c-96aa-4319-be26-96896272151d rhgb quiet noapic rd.luks.key=d8a28c4c-96aa-4319-be26-96896272151d=/keyfile:UUID=c47d962e-4be8-41d6-8216-8cf7a0d3b911 ipv6.disable=1

Procedure

1. List all parameters and their values.

   # sysctl -a

   Note

   The sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

2. To configure a parameter temporarily, enter:

   # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

   This sample changes the parameter value while the system is running. The changes take effect immediately and it does not require system reboot.

   Note

   The changes return back to default after your system reboots.

Procedure

1. List all parameters:

   # sysctl -a

   The command displays all kernel parameters that can be configured at runtime.

2. Configure a parameter permanently:

   # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

   The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

   Note

   To permanently modify kernel parameters, you can also make manual changes to the configuration files in the /etc/sysctl.d/ directory.

Procedure

1. Create a new configuration file in /etc/sysctl.d/:

   # vim /etc/sysctl.d/<some_file.conf>

2. Include kernel parameters, one per line:

   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

3. Save the configuration file.

4. Reboot the machine for the changes to take effect.

   • Alternatively, apply changes without rebooting:

     # sysctl -p /etc/sysctl.d/<some_file.conf>

     With this command, you can read values from the configuration file which you created earlier.

     See sysctl(8) and sysctl.d(5) man pages on your system for more information.

Procedure

1. Identify a kernel parameter you want to configure:

   # ls -l /proc/sys/<TUNABLE_CLASS>/

   The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

2. Assign a target value to the kernel parameter:

   # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

   The configuration changes applied by using a command are not permanent and will disappear after system reboot.

Verification

1. Verify the value of the newly set kernel parameter:

   # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configuring kernel settings
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure hugepages, packet size for loopback device, and limits on simultaneously open files.
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.kernel_settings
         vars:
           kernel_settings_sysctl:
             - name: fs.file-max
               value: 400000
             - name: kernel.threads-max
               value: 65536
           kernel_settings_sysfs:
             - name: /sys/class/net/lo/mtu
               value: 65000
           kernel_settings_transparent_hugepages: madvise
           kernel_settings_reboot_ok: true

   The settings specified in the example playbook include the following:

   kernel_settings_sysfs: <list_of_sysctl_settings>

   A YAML list of sysctl settings and the values you want to assign to these settings.

   kernel_settings_transparent_hugepages: <value>

   Controls the memory subsystem Transparent Huge Pages (THP) setting. You can disable THP support (never), enable it system wide (always) or inside MAD_HUGEPAGE regions (madvise).

   kernel_settings_reboot_ok: <true|false>

   The default is false. If set to true, the system role will determine if a reboot of the managed host is necessary for the requested changes to take effect and reboot it. If set to false, the role will return the variable kernel_settings_reboot_required with a value of true, indicating that a reboot is required. In this case, a user must reboot the managed node manually.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.kdump/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Copy the add-on file to the /boot/efi/loader/addons/ directory:

   # cp <my-addon>.addon.efi /boot/efi/loader/addons/

2. Reboot the system:

   # reboot

Procedure

1. Identify the running kernel version and machine ID:

   # uname -r
   # cat /etc/machine-id

2. Copy the add-on file to the specific directory associated with the UKI:

   # cp <my-addon>.addon.efi /boot/efi/EFI/Linux/<machine_id>-<kernel_version>.efi.extra.d/

3. Reboot the system:

   # reboot

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Update existing boot loader entries
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.bootloader
         vars:
           bootloader_settings:
             - kernel:
                 path: /boot/vmlinuz-6.12.0-0.el10_0.aarch64
               options:
                 - name: quiet
                   state: present
           bootloader_reboot_ok: true

   The settings specified in the example playbook include the following:

   kernel

   Specifies the kernel connected with the boot loader entry that you want to update.

   options

   Specifies the kernel command-line parameters to update for your chosen boot loader entry (kernel).

   bootloader_reboot_ok: true

   The role detects that a reboot is needed for the changes to take effect and performs a restart of the managed node.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create ~/vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      pwd: <password>

   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     vars_files:
       - ~/vault.yml
     tasks:
       - name: Set the bootloader password
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.bootloader
         vars:
           bootloader_password: "{{ pwd }}"
           bootloader_reboot_ok: true

   The settings specified in the example playbook include the following:

   bootloader_password: "{{ pwd }}"

   The variable ensures protection of boot parameters with a password.

   bootloader_reboot_ok: true

   The role detects that a reboot is needed for the changes to take effect and performs a restart of the managed node.

   Important

   Changing the boot loader password is not an idempotent transaction. This means that if you apply the same Ansible playbook again, the result will not be the same, and the state of the managed node will change.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --syntax-check --ask-vault-pass ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

Verification

1. On your managed node during the GRUB2 boot menu screen, press the e key for edit.

2. You will be prompted for a username and a password.

   Enter username: root

   The boot loader username is always root and you do not need to specify it in your Ansible playbook.

   Enter password: <password>

   The boot loader password corresponds to the pwd variable that you defined in the vault.yml file.

3. You can view or edit configuration of the particular boot loader entry.

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Update the boot loader timeout
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.bootloader
         vars:
           bootloader_timeout: 10

   The settings specified in the example playbook include the following:

   bootloader_timeout: 10

   Input an integer to control for how long the GRUB2 boot loader menu is displayed before booting the default entry.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. Remotely restart your managed node:

   # ansible managed-node-01.example.com -m ansible.builtin.reboot
   managed-node-01.example.com | CHANGED => {
       "changed": true,
       "elapsed": 21,
       "rebooted": true
   }

2. On the managed node, observe the GRUB2 boot menu screen.

   The highlighted entry will be executed automatically in 10s

   For how long this boot menu is displayed before GRUB2 automatically uses the default entry.

   • Alternative: you can remotely query for the "timeout" settings in the /boot/grub2/grub.cfg file of your managed node:

     # ansible managed-node-01.example.com -m ansible.builtin.command -a "grep 'timeout' /boot/grub2/grub.cfg"
     managed-node-01.example.com | CHANGED | rc=0 >>
     if [ x$feature_timeout_style = xy ] ; then
       set timeout_style=menu
       set timeout=10
     # Fallback normal timeout code in case the timeout_style feature is
       set timeout=10
     if [ x$feature_timeout_style = xy ] ; then
         set timeout_style=menu
         set timeout=10
         set orig_timeout_style=${timeout_style}
         set orig_timeout=${timeout}
           # timeout_style=menu + timeout=0 avoids the countdown code keypress check
           set timeout_style=menu
           set timeout=10
           set timeout_style=hidden
           set timeout=10
     if [ x$feature_timeout_style = xy ]; then
       if [ "${menu_show_once_timeout}" ]; then
         set timeout_style=menu
         set timeout=10
         unset menu_show_once_timeout
         save_env menu_show_once_timeout

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Gather information about the boot loader configuration
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.bootloader
         vars:
           bootloader_gather_facts: true
   
       - name: Display the collected boot loader configuration information
         debug:
           var: bootloader_facts

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
2. The kpatch module loads into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.

Procedure

1. Log in to the RHEL 10 web console.
2. Click Software Updates.

3. Check the status of your kernel patching settings.

   1. If the patching is not installed, click Install.

   2. To enable kernel patching, click Enable.

      

   3. Select the checkbox for applying kernel patches.
   4. Select whether you want to apply patches for current and future kernels or the current kernel only. If you decide to subscribe to applying patches for future kernels, the system also applies patches for the upcoming kernel releases.
   5. Click Apply.

Procedure

1. Optional: Check your kernel version:

   # uname -r
   6.12.0-55.9.1.el10_0.x86_64

2. Search for a live patching package that corresponds to the version of your kernel:

   # dnf search $(uname -r)

3. Install the live patching package:

   # dnf install kpatch

   This command installs and applies the latest cumulative live patches for that specific kernel only.

   If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-6_12_0-1-0-0.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optional: Check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            6.12.0-55.9.1.el10              @beaker-BaseOS
   kernel-core.x86_64            6.12.0-55.9.1.el10              @@commandline
   ...
   
   # uname -r
   6.12.0-55.9.1.el10_0.x86_64

2. Install the kpatch-dnf plugin:

   # dnf install kpatch-dnf

3. Enable automatic subscription to kernel live patches:

   # dnf kpatch auto
   Updating Subscription Management repositories.
   Last metadata expiration check: 1:38:21 ago on Fri 17 Sep 2021 07:29:53 AM EDT.
   Dependencies resolved.
   ==================================================
    Package                             Architecture
   ==================================================
   Installing:
    kpatch-patch-6_12_0-1               x86_64
    kpatch-patch-6_12_0-2               x86_64
   
   Transaction Summary
   ===================================================
   Install  2 Packages
   …​

   This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

   When you update the kernel, live patches are installed automatically during the new kernel installation process.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory that is loaded by the systemd system and service manager during future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-6_12_0-1-0-0.el10.x86_64.rpm.

   The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optional: Check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel

   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            6.12.0-0.el10              @beaker-BaseOS
   kernel-core.x86_64            6.12.0-0.el10              @@commandline
   ...

   # uname -r

   6.12.0-0.el10_0.x86_64

2. Disable automatic subscription to kernel live patches:

   # dnf kpatch manual

   Updating Subscription Management repositories.

   See kpatch(1) and dnf-kpatch(8) manual pages for more information.

Procedure

1. Select the live patching package:

   # dnf list installed | grep kpatch-patch

   kpatch-patch-6.12.0-0.el10_0.x86_64        0-0.el10        @@commandline
   ...

   The example output lists live patching packages that you installed.

2. Remove the live patching package:

   # dnf remove kpatch-patch-6.12.0-0.el10_0.x86_64

   When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

3. Reboot your system.

4. Verify the live patching package is removed:

   # dnf list installed | grep kpatch-patch

   The command displays no output if the package has been successfully removed.

Verification

1. Verify the kernel live patching solution is disabled:

   # kpatch list

   Loaded patch modules:

   The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Procedure

1. Select a kernel patch module:

   # kpatch list

   Loaded patch modules:
   kpatch_6_12_0_1_0_1 [enabled]
   
   Installed patch modules:
   kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)
   ...

2. Uninstall the selected kernel patch module.

   # kpatch uninstall kpatch_6_12_0_1_0_1

   uninstalling kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)

   • Note that the uninstalled kernel patch module is still loaded:

     # kpatch list

     Loaded patch modules:
     kpatch_6_12_0_1_0_1 [enabled]
     
     Installed patch modules:
     kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)

     When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

3. Reboot your system.

Verification

1. Verify that the kernel patch module is uninstalled:

   # kpatch list

   Loaded patch modules:
   ...

   This example output shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Procedure

1. Verify kpatch.service is enabled.

   # systemctl is-enabled kpatch.service

   enabled

2. Disable kpatch.service:

   # systemctl disable kpatch.service

   Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

   • Note that the applied kernel patch module is still loaded:

     # kpatch list

     Loaded patch modules:
     kpatch_6_12_0_1_0_1 [enabled]
     
     Installed patch modules:
     kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)

3. Reboot your system.

4. Optional: Verify the status of kpatch.service.

   # systemctl status kpatch.service

   ● kpatch.service - "Apply kpatch kernel patches"
      Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
      Active: inactive (dead)

   The example output testifies that kpatch.service is disabled. Thereby, the kernel live patching solution is not active.

5. Verify that the kernel patch module has been unloaded.

   # kpatch list

   Loaded patch modules:
   
   Installed patch modules:
   kpatch_6_12_0_1_0_1 (6.12.0.el10_0.x86_64)

   The example output shows that a kernel patch module is still installed but the kernel is not patched.

   Important

   Currently, Red Hat does not support reverting live patches without rebooting your system. In case of any issues, contact our support team.

Procedure

1. Display the memory statistics:

   # numastat -cm | egrep 'Node|Huge'

                    Node 0 Node 1 Node 2 Node 3  Total add
   AnonHugePages         0      2      0      8     10
   HugePages_Total       0      0      0      0      0
   HugePages_Free        0      0      0      0      0
   HugePages_Surp        0      0      0      0      0

2. Add the number of huge pages of a specified size to the node:

   # echo 20 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages

   Replace the following values:

   • 20 with the number of huge pages you want to reserve
   • 2048 with the size of the huge pages in KiB

   • node2 with the NUMA node on which you want to reserve the pages

     Note

     The directory name format in the path is hugepages-size_KiB where size is specified in KiB. The actual directory name might appear as hugepages-2048KiB or hugepages-2048kB depending on your system.

Procedure

1. Copy the default rule to the local override directory:

   cp /usr/lib/udev/rules.d/40-redhat-hotplug.rules /etc/udev/rules.d/

2. Edit the /etc/udev/rules.d/40-redhat-hotplug.rules file in a text editor and comment out the lines that handle memory hot-plug operations:

   # Memory hotadd request
   #SUBSYSTEM!="memory", GOTO="memory_hotplug_end"
   #ACTION!="add", GOTO="memory_hotplug_end"
   #CONST{arch}=="s390*", GOTO="memory_hotplug_end"
   #CONST{arch}=="ppc64*", GOTO="memory_hotplug_end"
   #ENV{.state}="online"
   #CONST{virt}=="none", ENV{.state}="online_movable"
   #ATTR{state}=="offline", ATTR{state}="$env{.state}"
   #LABEL="memory_hotplug_end"

3. Reload udev rules to apply the changes:

   udevadm control --reload-rules

4. Reboot the system, or remove the device and add it back to the system to ensure the new configuration takes effect.

Procedure

1. Identify the offline memory block identifier using the lsmem command:

   # lsmem

   RANGE                                  SIZE  STATE REMOVABLE BLOCK
   0x0000000000000000-0x000000007fffffff    2G online       yes  0-1
   0x0000000100000000-0x000000087fffffff   30G online       yes  4-33
   0x0000000880000000-0x00000008ffffffff    2G offline           34-41
   
   Memory block size:       256M
   Total online memory:      32G
   Total offline memory:      2G

   In this example, the offline memory blocks are 34-41.

2. Online a specific memory block by writing to its state file:

   echo online > /sys/devices/system/memory/memory<_identifier_>/state

   Replace memory<_identifier_> with the identifier of the memory block, for example, memory34.

1. Console log-level, defines the lowest priority of messages printed to the console.
2. Default log-level for messages without an explicit log-level attached to them.
3. Sets the lowest possible log-level configuration for the console log-level.

4. Sets default value for the console log-level at boot time.

   Each of these values defines a different rule for handling error messages.

Procedure

1. Install grub2-pc:

   # grub2-pc

2. Reinstall GRUB on the device where it is installed. For example, if sda is your device:

   # grub2-install /dev/sda

3. Reboot your system for the changes to take effect:

   # reboot

   See the grub2-pc(8) man page on your system for more information.

Procedure

1. Reinstall the grub2-efi and shim boot loader files:

   # dnf reinstall grub2-efi-x64 shim

2. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. List the disk partition that stores GRUB:

   # bootlist -m normal -o

   sda1

2. Reinstall GRUB on the disk partition:

   # grub2-install partition

   Replace partition with the identified GRUB partition, such as /dev/sda1.

3. Reboot your system for the changes to take effect:

   # reboot

   See the grub-install(1) man page on your system for more information.

Procedure

1. Remove the configuration files:

   # rm /etc/grub.d/*
   # rm /etc/sysconfig/grub

2. Reinstall packages.

   • On BIOS-based machines:

     # dnf reinstall grub2-pc grub2-tools

   • On UEFI-based machines:

     # dnf reinstall grub2-efi shim grub2-tools grub2-common

3. Rebuild the grub.cfg file for the changes to take effect:

   # grub2-mkconfig -o /boot/grub2/grub.cfg

   This applies to both, BIOS and UEFI based systems.

   Warning

   The path to rebuild grub.cfg is same for both BIOS and UEFI based machines. Actual grub.cfg is present at BIOS path only. The UEFI path has a stub file that must not be modified or recreated using grub2-mkconfig command.

4. Follow Reinstalling GRUB procedure to restore GRUB on the /boot/ partition.

Procedure

1. On the Anaconda installer, click KDUMP and enable kdump.
2. In Kdump Memory Reservation, select Manual` if you must customize the memory reserve.
3. In KDUMP > Memory To Be Reserved (MB), set the required memory reserve for kdump.

Procedure

1. Check if kdump is installed on your system:

   # rpm -q kexec-tools kdump-utils makedumpfile

2. If the kdump is not installed, install required packages:

   # dnf install kexec-tools kdump-utils makedumpfile

Procedure

1. Configure the default value for crash kernel:

   # kdumpctl reset-crashkernel --kernel=ALL

   When configuring the crashkernel= value, test the configuration by rebooting the system with kdump enabled. If the kdump kernel fails to boot, increase the memory size gradually to set an acceptable value.

2. To use a custom crashkernel= value:

   1. Configure the required memory reserve.

      crashkernel=192M

      Optionally, you can set the amount of reserved memory to a variable depending on the total amount of installed memory by using the syntax crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

      crashkernel=1G-4G:192M,2G-64G:256M

      The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for kdump.

   2. Optional: Offset the reserved memory.

      Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

      crashkernel=192M@16M

      The example reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If you offset to 0 or do not specify a value, kdump offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example, crashkernel=1G-4G:192M,2G-64G:256M@16M.

   3. Update the boot loader configuration:

      # grubby --update-kernel ALL --args "crashkernel=<custom-value>"

      The <custom-value> must contain the custom crashkernel= value that you have configured for the crash kernel.

3. Reboot for changes to take effect:

   # reboot

1. Activate the sysrq key to boot into the kdump kernel:

   # echo c > /proc/sysrq-trigger

   The command causes kernel to crash and reboots the kernel if required.

2. Display the /etc/kdump.conf file and check if the vmcore file is saved in the target destination.

   See the grubby(8) man page on your system for more information.

Procedure

1. As a root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.

2. Enter the following command to enable crash dump file compression:

   core_collector makedumpfile -l --message-level 1 -d 31

   The -l option sets the compressed file format to LZO. The -d option sets the dump level to 31. The --message-level option sets the message level to 1. You can also use the -c, -p, or -z options to specify other compression formats.

   See makedumpfile(8) man page on your system for more information.

Procedure

1. As a root user, remove the hash sign (#) from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a required action.

   failure_action poweroff

Procedure

1. Enable the kdump service:

   # kdumpctl restart

2. Check the status of the kdump service with the kdumpctl:

   # kdumpctl status

   kdump:Kdump is operational

   Optionally, if you use the systemctl command, the output prints in the systemd journal.

3. Start a kernel crash to test the kdump configuration. The sysrq-trigger key combination causes the kernel to crash and might reboot the system if required.

   # echo c > /proc/sysrq-trigger

   On a kernel reboot, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file. The default is /var/crash/.

Procedure

1. Enable the kdump service for multi-user.target:

   # systemctl enable kdump.service

2. Start the service in the current session:

   # systemctl start kdump.service

3. Stop the kdump service:

   # systemctl stop kdump.service

4. Disable the kdump service:

   # systemctl disable kdump.service

   Warning

   It is advisable to set kptr_restrict=1 as default. When kptr_restrict is set to (1) as default, the kdumpctl service loads the crash kernel regardless of Kernel Address Space Layout (KASLR) is enabled.

   If kptr_restrict is not set to 1 and KASLR is enabled, the contents of /proc/kore file are generated as all zeros. The kdumpctl service fails to access the /proc/kcore file and load the crash kernel. The kexec-kdump-howto.txt file displays a warning message to set kptr_restrict=1. Verify for the following in the sysctl.conf file to ensure that kdumpctl service loads the crash kernel:

   • Kernel kptr_restrict=1 in the sysctl.conf file.

Procedure

1. Display the list of modules that are loaded to the currently running kernel. Select the kernel module that you intend to block from loading:

   $ lsmod

   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1

2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file. For example:

   KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

   Also, consider the following example by using the modprobe.blacklist=<modules> configuration option:

   KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

3. Restart the kdump service:

   # systemctl restart kdump

   See the dracut.cmdline man page on your system for more information.

Procedure

1. Print the estimate crashkernel= value:

   # *kdumpctl estimate*

   Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
      Reserved crashkernel:    256M
      Recommended crashkernel: 652M
   
      Kernel image size:   47M
      Kernel modules size: 8M
      Initramfs size:      20M
      Runtime reservation: 64M
      LUKS required size:  512M
      Large modules: <none>
      WARNING: Current crashkernel size is lower than recommended size 652M.

2. Configure the amount of required memory by increasing the crashkernel= value.

3. Reboot the system.

   Note

   If the kdump service still fails to save the dump file to the encrypted target, increase the crashkernel= value as required.

Procedure

1. Add the crashkernel= command-line parameter to all installed kernels:

   # grubby --update-kernel=ALL --args="crashkernel=xxM"

   xxM is the required memory in megabytes.

2. In terms of UKI, add the crashkernel= command-line parameter to all installed UKIs by copying an add-on associated with the required argument from /lib/modules/$(uname -r)/vmlinuz-virt.efi.extra.d/ to /boot/efi/loader/addons/.

3. Reboot the system:

   # reboot

4. Enable the kdump service:

   # systemctl enable --now kdump.service

Procedure

1. List the kernels installed on the machine:

   # ls -a /boot/vmlinuz-*

   /boot/vmlinuz-0-rescue-2930657cd0dc43c2b75db480e5e5b4a9
   /boot/vmlinuz-6.12.0-55.9.1.el10_0.x86_64
   /boot/vmlinuz-6.12.0-55.9.1.el10_0.x86_64

2. Add a specific kdump kernel to the system’s Grand Unified Boot Loader (GRUB) configuration:

   For example:

   # grubby --update-kernel=vmlinuz-6.12.0-55.9.1.el10_0.x86_64 --args="crashkernel=xxM"

   xxM is the required memory reserve in megabytes.

3. In terms of UKI, add the crashkernel= command-line parameter to a specific UKI by copying an add-on associated with the required argument from /lib/modules/$(uname -r)/vmlinuz-virt.efi.extra.d/ to /boot/efi/EFI/Linux/<machine-id>-<kernel-version>.efi.extra.d/.

4. Enable the kdump service:

   # systemctl enable --now kdump.service

Procedure

1. To stop the kdump service in the current session:

   # systemctl stop kdump.service

2. To disable the kdump service:

   # systemctl disable kdump.service

   Warning

   You must set kptr_restrict=1 as default. When kptr_restrict is set to (1) as default, the kdumpctl service loads the crash kernel regardless of whether the Kernel Address Space Layout (KASLR) is enabled.

   If kptr_restrict is not set to 1 and KASLR is enabled, the contents of /proc/kore file are generated as all zeros. The kdumpctl service fails to access the /proc/kcore file and load the crash kernel. The kexec-kdump-howto.txt file displays a warning message, which suggest you to set kptr_restrict=1. Verify for the following in the sysctl.conf file to ensure that kdumpctl service loads the crash kernel:

   • Kernel kptr_restrict=1 in the sysctl.conf file.

Procedure

1. To configure final_action, edit the /etc/kdump.conf file and add one of the following options:

   • final_action reboot
   • final_action halt
   • final_action poweroff

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure

1. To configure an action to take if the dump fails, edit the /etc/kdump.conf file and specify one of the failure_action options:

   • failure_action reboot
   • failure_action halt
   • failure_action poweroff
   • failure_action shell
   • failure_action dump_to_rootfs

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure

1. Install the kexec-tools, kdump-utils, and makedumpfile packages.

2. Configure the default value for crashkernel:

   # kdumpctl reset-crashkernel --fadump=on --kernel=ALL

3. Optional: Reserve boot memory instead of the default value:

   # grubby --update-kernel ALL --args="fadump=on crashkernel=xxM"

   xxM is the required memory size in megabytes.

   Note

   When specifying boot configuration options, test the configurations by rebooting the kernel with kdump enabled. If the kdump kernel fails to boot, increase the crashkernel value gradually to set an appropriate value.

4. Reboot for changes to take effect:

   # reboot

Procedure

1. Add or edit the following lines in the /etc/sysctl.conf file to ensure that kdump starts as expected for sadump:

   kernel.panic=0
   kernel.unknown_nmi_panic=1

   Warning

   In particular, ensure that after kdump, the system does not reboot. If the system reboots after kdump has failed to save the vmcore file, then it is not possible to start the sadump.

2. Set the failure_action parameter in /etc/kdump.conf appropriately as halt or shell.

   failure_action shell

   See the FUJITSU Server PRIMEQUEST 2000 Series Installation Manual for more information.

Procedure

1. Enable the relevant repositories:

   # subscription-manager repos --enable baseos repository

   # subscription-manager repos --enable appstream repository

   # subscription-manager repos --enable rhel-10-for-x86_64-baseos-debug-rpms

2. Install the crash package:

   # dnf install crash

3. Install the kernel-debuginfo package:

   # dnf install kernel-debuginfo

   The package kernel-debuginfo corresponds to the running kernel and provides the data necessary for the dump analysis.