Managing, monitoring, and updating the kernel

Procedure

1. List the available versions of a package:

   $ dnf repoquery <package_name>

2. Display the list of files in a package:

   $ dnf repoquery -l <package_name>

Procedure

1. To update the kernel, enter the following command:

   # dnf update kernel

   This command updates the kernel along with all dependencies to the latest available version.

2. Reboot your system for the changes to take effect.

Procedure

1. Select a kernel module you want to load.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Load the relevant kernel module:

   # modprobe <MODULE_NAME>

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. List all loaded kernel modules:

   # lsmod

2. Select the kernel module you want to unload.

   If a kernel module has dependencies, unload those prior to unloading the kernel module. For details on identifying modules with dependencies, see Listing currently loaded kernel modules and Kernel module dependencies.

3. Unload the relevant kernel module:

   # modprobe -r <MODULE_NAME>

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. Boot the system into the boot loader.
2. Use the cursor keys to highlight the relevant boot loader entry.

3. Press the e key to edit the entry.

   Figure 3.1. Kernel boot menu

   
4. Use the cursor keys to navigate to the line that starts with linux.

5. Append modprobe.blacklist=module_name to the end of the line.

   Figure 3.2. Kernel boot entry

   

   The serio_raw kernel module illustrates a rogue module to be unloaded early in the boot process.

6. Press Ctrl+X to boot using the modified configuration.

Procedure

1. Select a kernel module you want to load during the boot process.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for the module:

   # echo <MODULE_NAME> > /etc/modules-load.d/<MODULE_NAME>.conf

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Verification

1. After reboot, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   The example command above should succeed and display the relevant kernel module.

Procedure

1. List modules loaded to the currently running kernel by using the lsmod command:

   $ lsmod
   Module                  Size  Used by
   tls                   131072  0
   uinput                 20480  1
   snd_seq_dummy          16384  0
   snd_hrtimer            16384  1
   …

   In the output, identify the module you want to prevent from being loaded.

   • Alternatively, identify an unloaded kernel module you want to prevent from potentially loading in the /lib/modules/<KERNEL-VERSION>/kernel/<SUBSYSTEM>/ directory, for example:

     $ ls /lib/modules/4.18.0-477.20.1.el8_8.x86_64/kernel/crypto/
     ansi_cprng.ko.xz        chacha20poly1305.ko.xz  md4.ko.xz               serpent_generic.ko.xz
     anubis.ko.xz            cmac.ko.xz…

2. Create a configuration file serving as a denylist:

   # touch /etc/modprobe.d/denylist.conf

3. In a text editor of your choice, combine the names of modules you want to exclude from automatic loading to the kernel with the blacklist configuration command, for example:

   # Prevents <KERNEL-MODULE-1> from being loaded
   blacklist <MODULE-NAME-1>
   install <MODULE-NAME-1> /bin/false
   
   # Prevents <KERNEL-MODULE-2> from being loaded
   blacklist <MODULE-NAME-2>
   install <MODULE-NAME-2> /bin/false
   …

   Because the blacklist command does not prevent the module from being loaded as a dependency for another kernel module that is not in a denylist, you must also define the install line. In this case, the system runs /bin/false instead of installing the module. The lines starting with a hash sign are comments you can use to make the file more readable.

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

4. Create a backup copy of the current initial RAM disk image before rebuilding:

   # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

   • Alternatively, create a backup copy of an initial RAM disk image which corresponds to the kernel version for which you want to prevent kernel modules from automatic loading:

     # cp /boot/initramfs-<VERSION>.img /boot/initramfs-<VERSION>.img.bak.$(date +%m-%d-%H%M%S)

5. Generate a new initial RAM disk image to apply the changes:

   # dracut -f -v

   • If you build an initial RAM disk image for a different kernel version than your system currently uses, specify both target initramfs and kernel version:

     # dracut -f -v /boot/initramfs-<TARGET-VERSION>.img <CORRESPONDING-TARGET-KERNEL-VERSION>

6. Restart the system:

   $ reboot

Procedure

1. Create the /root/testmodule/test.c file with the following content.

   #include <linux/module.h>
   #include <linux/kernel.h>
   
   int init_module(void)
       { printk("Hello World\n This is a test\n"); return 0; }
   
   void cleanup_module(void)
       { printk("Good Bye World"); }
   
   MODULE_LICENSE("GPL");

   The test.c file is a source file that provides the main functionality to the kernel module. The file has been created in a dedicated /root/testmodule/ directory for organizational purposes. After the module compilation, the /root/testmodule/ directory will contain multiple files.

   The test.c file includes from the system libraries:

   • The linux/kernel.h header file is necessary for the printk() function in the example code.
   • The linux/module.h file contains function declarations and macro definitions to be shared between several source files written in C programming language.
2. Follow the init_module() and cleanup_module() functions to start and end the kernel logging function printk(), which prints text.

3. Create the /root/testmodule/Makefile file with the following content.

   obj-m := test.o

   The Makefile contains instructions that the compiler has to produce an object file specifically named test.o. The obj-m directive specifies that the resulting test.ko file is going to be compiled as a loadable kernel module. Alternatively, the obj-y directive would instruct to build test.ko as a built-in kernel module.

4. Compile the kernel module.

   # make -C /lib/modules/$(uname -r)/build M=/root/testmodule modules
   make: Entering directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'
     CC [M]  /root/testmodule/test.o
     MODPOST /root/testmodule/Module.symvers
     CC [M]  /root/testmodule/test.mod.o
     LD [M]  /root/testmodule/test.ko
     BTF [M] /root/testmodule/test.ko
   Skipping BTF generation for /root/testmodule/test.ko due to unavailability of vmlinux
   make: Leaving directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'

   The compiler creates an object file (test.o) for each source file (test.c) as an intermediate step before linking them together into the final kernel module (test.ko).

   After a successful compilation, /root/testmodule/ contains additional files that relate to the compiled custom kernel module. The compiled module itself is represented by the test.ko file.

Verification

1. Optional: check the contents of the /root/testmodule/ directory:

   # ls -l /root/testmodule/
   total 152
   -rw-r—​r--. 1 root root    16 Jul 26 08:19 Makefile
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 modules.order
   -rw-r—​r--. 1 root root     0 Jul 26 08:20 Module.symvers
   -rw-r—​r--. 1 root root   224 Jul 26 08:18 test.c
   -rw-r—​r--. 1 root root 62176 Jul 26 08:20 test.ko
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 test.mod
   -rw-r—​r--. 1 root root   849 Jul 26 08:20 test.mod.c
   -rw-r—​r--. 1 root root 50936 Jul 26 08:20 test.mod.o
   -rw-r—​r--. 1 root root 12912 Jul 26 08:20 test.o

2. Copy the kernel module to the /lib/modules/$(uname -r)/ directory:

   # cp /root/testmodule/test.ko /lib/modules/$(uname -r)/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v test
   insmod /lib/modules/5.14.0-1.el9.x86_64/test.ko

5. Verify that the kernel module was successfully loaded:

   # lsmod | grep test
   test                   16384  0

6. Read the latest messages from the kernel ring buffer:

   # dmesg
   [74422.545004] Hello World
                   This is a test

Procedure

1. Boot into the GRUB 2 boot menu.
2. Select the kernel you want to start.
3. Press the e key to edit the kernel parameters.
4. Find the kernel command line by moving the cursor down. The kernel command line starts with linux on 64-Bit IBM Power Series and x86-64 BIOS-based systems, or linuxefi on UEFI systems.

5. Move the cursor to the end of the line.

   Note

   Press Ctrl+a to jump to the start of the line and Ctrl+e to jump to the end of the line. On some systems, Home and End keys might also work.

6. Edit the kernel parameters as required. For example, to run the system in emergency mode, add the emergency parameter at the end of the linux line:

   linux   ($root)/vmlinuz-5.14.0-63.el9.x86_64 root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet emergency

   To enable the system messages, remove the rhgb and quiet parameters.

7. Press Ctrl+x to boot with the selected kernel and the modified command line parameters.

Procedure

1. Add the following two lines to the /etc/default/grub file:

   GRUB_TERMINAL="serial"
   GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

   The first line disables the graphical terminal. The GRUB_TERMINAL key overrides values of GRUB_TERMINAL_INPUT and GRUB_TERMINAL_OUTPUT keys.

   The second line adjusts the baud rate (--speed), parity and other values to fit your environment and hardware. Note that a much higher baud rate, for example 115200, is preferable for tasks such as following log files.

2. Update the GRUB configuration file.

   • On BIOS-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

3. Reboot the system for the changes to take effect.

Procedure

1. Add or remove a kernel parameter for individual kernels in a post installation script with grubby:

   # grubby --update-kernel <PATH_TO_KERNEL> --args "<NEW_ARGUMENTS>"

   For example, add the noapic parameter to the chosen kernel:

   # grubby --update-kernel /boot/vmlinuz-5.14.0-362.8.1.el9_3.x86_64 --args "noapic"

   The parameter is propagated into the BLS snippets, but not into the /etc/default/grub file.

2. Overwrite BLS snippets with the contents of the GRUB_CMDLINE_LINUX values present in the /etc/default/grub file:

   # grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
   Generating grub configuration file …​
   Adding boot menu entry for UEFI Firmware Settings …​
   done

   Note

   Other changes, such as changes made to GRUB_TIMEOUT key (also included in the /etc/default/grub GRUB configuration file), do get propagated to the new grub.cfg by default.

Verification

1. Reboot your operating system.

2. Verify that the parameters are included in the /proc/cmdline file.

   For example, /proc/cmdline contains the noapic kernel parameter:

   BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-425.3.1.el8.x86_64 root=/dev/mapper/RHELCSB-Root ro vconsole.keymap=us crashkernel=auto rd.lvm.lv=RHELCSB/Root rd.luks.uuid=luks-d8a28c4c-96aa-4319-be26-96896272151d rhgb quiet noapic rd.luks.key=d8a28c4c-96aa-4319-be26-96896272151d=/keyfile:UUID=c47d962e-4be8-41d6-8216-8cf7a0d3b911 ipv6.disable=1

Procedure

1. List all parameters and their values.

   # sysctl -a

   Note

   The # sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

2. To configure a parameter temporarily, enter:

   # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

   The sample command above changes the parameter value while the system is running. The changes take effect immediately, without a need for restart.

   Note

   The changes return back to default after your system reboots.

Procedure

1. List all parameters.

   # sysctl -a

   The command displays all kernel parameters that can be configured at runtime.

2. Configure a parameter permanently:

   # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

   The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

Procedure

1. Create a new configuration file in /etc/sysctl.d/.

   # vim /etc/sysctl.d/<some_file.conf>

2. Include kernel parameters, one per line.

   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

3. Save the configuration file.

4. Reboot the machine for the changes to take effect.

   • Alternatively, to apply changes without rebooting, enter:

     # sysctl -p /etc/sysctl.d/<some_file.conf>

     The command enables you to read values from the configuration file, which you created earlier.

Procedure

1. Identify a kernel parameter you want to configure.

   # ls -l /proc/sys/<TUNABLE_CLASS>/

   The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

2. Assign a target value to the kernel parameter.

   # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

   The command makes configuration changes that will disappear once the system is restarted.

Verification

1. Verify the value of the newly set kernel parameter.

   # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configuring kernel settings
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure hugepages, packet size for loopback device, and limits on simultaneously open files.
         ansible.builtin.include_role:
           name: rhel-system-roles.kernel_settings
         vars:
           kernel_settings_sysctl:
             - name: fs.file-max
               value: 400000
             - name: kernel.threads-max
               value: 65536
           kernel_settings_sysfs:
             - name: /sys/class/net/lo/mtu
               value: 65000
           kernel_settings_transparent_hugepages: madvise
           kernel_settings_reboot_ok: true

   The settings specified in the example playbook include the following:

   kernel_settings_sysfs: <list_of_sysctl_settings>

   A YAML list of sysctl settings and the values you want to assign to these settings.

   kernel_settings_transparent_hugepages: <value>

   Controls the memory subsystem Transparent Huge Pages (THP) setting. You can disable THP support (never), enable it system wide (always) or inside MAD_HUGEPAGE regions (madvise).

   kernel_settings_reboot_ok: <true|false>

   The default is false. If set to true, the system role will determine if a reboot of the managed host is necessary for the requested changes to take effect and reboot it. If set to false, the role will return the variable kernel_settings_reboot_required with a value of true, indicating that a reboot is required. In this case, a user must reboot the managed node manually.

1. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

2. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Update existing boot loader entries
         ansible.builtin.include_role:
           name: rhel-system-roles.bootloader
         vars:
           bootloader_settings:
             - kernel:
                 path: /boot/vmlinuz-5.14.0-362.24.1.el9_3.aarch64
               options:
                 - name: quiet
                   state: present
           bootloader_reboot_ok: true

   The settings specified in the example playbook include the following:

   kernel

   Specifies the kernel connected with the boot loader entry that you want to update.

   options

   Specifies the kernel command-line parameters to update for your chosen boot loader entry (kernel).

   bootloader_reboot_ok: true

   The role detects that a reboot is needed for the changes to take effect and performs a restart of the managed node.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      pwd: <password>

   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     vars_files:
       - vault.yml
     tasks:
       - name: Set the bootloader password
         ansible.builtin.include_role:
           name: rhel-system-roles.bootloader
         vars:
           bootloader_password: "{{ pwd }}"
           bootloader_reboot_ok: true

   The settings specified in the example playbook include the following:

   bootloader_password: "{{ pwd }}"

   The variable ensures protection of boot parameters with a password.

   bootloader_reboot_ok: true

   The role detects that a reboot is needed for the changes to take effect and performs a restart of the managed node.

   Important

   Changing the boot loader password is not an idempotent transaction. This means that if you apply the same Ansible playbook again, the result will not be the same, and the state of the managed node will change.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --syntax-check --ask-vault-pass ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

Verification

1. On your managed node during the GRUB2 boot menu screen, press the e key for edit.

   

2. You will be prompted for a username and a password:

   

   Enter username: root

   The boot loader username is always root and you do not need to specify it in your Ansible playbook.

   Enter password: <password>

   The boot loader password corresponds to the pwd variable that you defined in the vault.yml file.

3. You can view or edit configuration of the particular boot loader entry:

   

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Update the boot loader timeout
         ansible.builtin.include_role:
           name: rhel-system-roles.bootloader
         vars:
           bootloader_timeout: 10

   The settings specified in the example playbook include the following:

   bootloader_timeout: 10

   Input an integer to control for how long the GRUB2 boot loader menu is displayed before booting the default entry.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. Remotely restart your managed node:

   # ansible managed-node-01.example.com -m ansible.builtin.reboot
   managed-node-01.example.com | CHANGED => {
       "changed": true,
       "elapsed": 21,
       "rebooted": true
   }

2. On the managed node, observe the GRUB2 boot menu screen.

   

   The highlighted entry will be executed automatically in 10s

   For how long this boot menu is displayed before GRUB2 automatically uses the default entry.

   • Alternative: you can remotely query for the "timeout" settings in the /boot/grub2/grub.cfg file of your managed node:

     # ansible managed-node-01.example.com -m ansible.builtin.command -a "grep 'timeout' /boot/grub2/grub.cfg"
     managed-node-01.example.com | CHANGED | rc=0 >>
     if [ x$feature_timeout_style = xy ] ; then
       set timeout_style=menu
       set timeout=10
     # Fallback normal timeout code in case the timeout_style feature is
       set timeout=10
     if [ x$feature_timeout_style = xy ] ; then
         set timeout_style=menu
         set timeout=10
         set orig_timeout_style=${timeout_style}
         set orig_timeout=${timeout}
           # timeout_style=menu + timeout=0 avoids the countdown code keypress check
           set timeout_style=menu
           set timeout=10
           set timeout_style=hidden
           set timeout=10
     if [ x$feature_timeout_style = xy ]; then
       if [ "${menu_show_once_timeout}" ]; then
         set timeout_style=menu
         set timeout=10
         unset menu_show_once_timeout
         save_env menu_show_once_timeout

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configuration and management of GRUB2 boot loader
     hosts: managed-node-01.example.com
     tasks:
       - name: Gather information about the boot loader configuration
         ansible.builtin.include_role:
           name: rhel-system-roles.bootloader
         vars:
           bootloader_gather_facts: true
   
       - name: Display the collected boot loader configuration information
         debug:
           var: bootloader_facts

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.bootloader/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Procedure

1. Optional: Check your kernel version:

   # uname -r
   5.14.0-1.el9.x86_64

2. Search for a live patching package that corresponds to the version of your kernel:

   # dnf search $(uname -r)

3. Install the live patching package:

   # dnf install "kpatch-patch = $(uname -r)"

   The command above installs and applies the latest cumulative live patches for that specific kernel only.

   If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optional: Check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Install the kpatch-dnf plugin:

   # dnf install kpatch-dnf

3. Enable automatic subscription to kernel live patches:

   # dnf kpatch auto
   Updating Subscription Management repositories.
   Last metadata expiration check: 1:38:21 ago on Fri 17 Sep 2021 07:29:53 AM EDT.
   Dependencies resolved.
   ==================================================
    Package                             Architecture
   ==================================================
   Installing:
    kpatch-patch-5_14_0-1               x86_64
    kpatch-patch-5_14_0-2               x86_64
   
   Transaction Summary
   ===================================================
   Install  2 Packages
   …​

   This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

   In the future, when you update the kernel, live patches will automatically be installed during the new kernel installation process.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.el9.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optional: Check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Disable automatic subscription to kernel live patches:

   # dnf kpatch manual
   Updating Subscription Management repositories.

Procedure

1. Select the live patching package.

   # dnf list installed | grep kpatch-patch
   kpatch-patch-5_14_0-1.x86_64        0-1.el9        @@commandline
   …​

   The example output above lists live patching packages that you installed.

2. Remove the live patching package.

   # dnf remove kpatch-patch-5_14_0-1.x86_64

   When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

3. Reboot your system.

4. Verify that the live patching package has been removed.

   # dnf list installed | grep kpatch-patch

   The command displays no output if the package has been successfully removed.

Verification

1. Verify that the kernel live patching solution is disabled.

   # kpatch list
   Loaded patch modules:

   The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Procedure

1. Select a kernel patch module:

   # kpatch list
   Loaded patch modules:
   kpatch_5_14_0_1_0_1 [enabled]
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)
   …​

2. Uninstall the selected kernel patch module.

   # kpatch uninstall kpatch_5_14_0_1_0_1
   uninstalling kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   • Note that the uninstalled kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     <NO_RESULT>

     When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

3. Reboot your system.

Verification

1. Verify that the kernel patch module is uninstalled.

   # kpatch list
   Loaded patch modules:
   …​

   This example output shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Procedure

1. Verify kpatch.service is enabled.

   # systemctl is-enabled kpatch.service
   enabled

2. Disable kpatch.service:

   # systemctl disable kpatch.service
   Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

   • Note that the applied kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

3. Reboot your system.

4. Optional: Verify the status of kpatch.service.

   # systemctl status kpatch.service
   ● kpatch.service - "Apply kpatch kernel patches"
      Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
      Active: inactive (dead)

   The example output testifies that kpatch.service has been disabled and is not running. Thereby, the kernel live patching solution is not active.

5. Verify that the kernel patch module has been unloaded.

   # kpatch list
   Loaded patch modules:
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   The example output above shows that a kernel patch module is still installed but the kernel is not patched.

1. Console log-level, defines the lowest priority of messages printed to the console.
2. Default log-level for messages without an explicit log-level attached to them.
3. Sets the lowest possible log-level configuration for the console log-level.

4. Sets default value for the console log-level at boot time.

   Each of these values above defines a different rule for handling error messages.

Procedure

1. Reinstall GRUB on the device where it is installed. For example, if sda is your device:

   # grub2-install /dev/sda

2. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Reinstall the grub2-efi and shim boot loader files:

   # yum reinstall grub2-efi shim

2. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Determine the disk partition that stores GRUB:

   # bootlist -m normal -o
   sda1

2. Reinstall GRUB on the disk partition:

   # grub2-install partition

   Replace partition with the GRUB partition that you found in the previous step, such as /dev/sda1.

3. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Remove the configuration files.

   # rm /etc/grub.d/*
   # rm /etc/sysconfig/grub

2. Reinstall packages.

   • On BIOS-based machines, enter:

     # yum reinstall grub2-tools

   • On UEFI-based machines, enter:

     # yum reinstall grub2-efi shim grub2-tools

3. Rebuild the grub.cfg file for the changes to take effect.

   • On BIOS-based machines, enter:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines, enter:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

4. Follow Reinstalling GRUB procedure to restore GRUB on the /boot/ partition.

Procedure

1. On the Anaconda installer, click KDUMP and enable kdump:
2. In Kdump Memory Reservation, select Manual` if you must customize the memory reserve.

3. In KDUMP > Memory To Be Reserved (MB), set the required memory reserve for kdump.

   = Installing kdump on the command line

Procedure

1. Check if kdump is installed on your system:

   # rpm -q kexec-tools

   Output if the package is installed:

   # kexec-tools-2.0.22-13.el9.x86_64

   Output if the package is not installed:

   package kexec-tools is not installed

2. Install kdump and other necessary packages:

   # dnf install kexec-tools

Procedure

1. Configure the default value for crash kernel:

   # kdumpctl reset-crashkernel --kernel=ALL

   When configuring the crashkernel= value, test the configuration by rebooting the system with kdump enabled. If the kdump kernel fails to boot, increase the memory size gradually to set an acceptable value.

2. To use a custom crashkernel= value:

   1. Configure the required memory reserve.

      crashkernel=192M

      Optionally, you can set the amount of reserved memory to a variable depending on the total amount of installed memory by using the syntax crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

      crashkernel=1G-4G:192M,2G-64G:256M

      The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for kdump.

   2. Optional: Offset the reserved memory.

      Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

      crashkernel=192M@16M

      The example reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If you offset to 0 or do not specify a value, kdump offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example, crashkernel=1G-4G:192M,2G-64G:256M@16M.

   3. Update the bootloader configuration:

      # grubby --update-kernel ALL --args "crashkernel=<custom-value>"

      The <custom-value> must contain the custom crashkernel= value that you have configured for the crash kernel.

3. Reboot for changes to take effect:

   # reboot

1. Activate the sysrq key to boot into the kdump kernel:

   # echo c > /proc/sysrq-trigger

   The command causes kernel to crash and reboots the kernel if required.

2. Display the /etc/kdump.conf file and check if the vmcore file is saved in the target destination.

Procedure

1. As a root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.
2. Enter the following command to enable crash dump file compression:

Procedure

1. As a root user, remove the hash sign (#) from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a required action.

   failure_action poweroff

Procedure

1. Enable the kdump service:

   # kdumpctl restart

2. Check the status of the kdump service with the kdumpctl:

   # kdumpctl status
     kdump:Kdump is operational

   Optionally, if you use the systemctl command, the output prints in the systemd journal.

3. Start a kernel crash to test the kdump configuration. The sysrq-trigger key combination causes the kernel to crash and might reboot the system if required.

   # echo c > /proc/sysrq-trigger

   On a kernel reboot, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file. The default is /var/crash/.

Procedure

1. Enable the kdump service for multi-user.target:

   # systemctl enable kdump.service

2. Start the service in the current session:

   # systemctl start kdump.service

3. Stop the kdump service:

   # systemctl stop kdump.service

4. Disable the kdump service:

   # systemctl disable kdump.service

Procedure

1. Display the list of modules that are loaded to the currently running kernel. Select the kernel module that you intend to block from loading:

   $ lsmod
   
   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1

2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file. For example:

   KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

   Also, consider the following example by using the modprobe.blacklist=<modules> configuration option:

   KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

3. Restart the kdump service:

   # systemctl restart kdump

Procedure

1. Print the estimate crashkernel= value:

   # *kdumpctl estimate*
   
   Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
      Reserved crashkernel:    256M
      Recommended crashkernel: 652M
   
      Kernel image size:   47M
      Kernel modules size: 8M
      Initramfs size:      20M
      Runtime reservation: 64M
      LUKS required size:  512M
      Large modules: <none>
      WARNING: Current crashkernel size is lower than recommended size 652M.

2. Configure the amount of required memory by increasing the crashkernel= value.
3. Reboot the system.

Procedure

1. In the web console, open the Kernel dump tab and start the kdump service by setting the Kernel crash dump switch to on.

2. Configure the kdump memory usage in the terminal, for example:

   $ sudo grubby --update-kernel ALL --args crashkernel=512M

   Restart the system to apply the changes.

3. In the Kernel dump tab, click Edit at the end of the Crash dump location field.

4. Specify the target directory for saving the vmcore dump file:

   • For a local filesystem, select Local Filesystem from the drop-down menu.

   • For a remote system by using the SSH protocol, select Remote over SSH from the drop-down menu and specify the following fields:

     • In the Server field, enter the remote server address.
     • In the SSH key field, enter the SSH key location.
     • In the Directory field, enter the target directory.

   • For a remote system by using the NFS protocol, select Remote over NFS from the drop-down menu and specify the following fields:

     • In the Server field, enter the remote server address.
     • In the Export field, enter the location of the shared folder of an NFS server.

     • In the Directory field, enter the target directory.

       Note

       You can reduce the size of the vmcore file by selecting the Compression checkbox.

5. Optional: Display the automation script by clicking View automation script.

   A window with the generated script opens. You can browse a shell script and an Ansible playbook generation options tab.

6. Optional: Copy the script by clicking Copy to clipboard.

   You can use this script to apply the same configuration on multiple machines.

Verification

1. Click Test configuration.

2. Click Crash system under Test kdump settings.

   Warning

   When you start the system crash, the kernel operation stops and results in a system crash with data loss.

Procedure

1. Add the crashkernel= command-line parameter to all installed kernels.

   # grubby --update-kernel=ALL --args="crashkernel=xxM"

   xxM is the required memory in megabytes.

2. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. List the kernels installed on the machine.

   # ls -a /boot/vmlinuz-*
   /boot/vmlinuz-0-rescue-2930657cd0dc43c2b75db480e5e5b4a9
   /boot/vmlinuz-4.18.0-330.el8.x86_64
   /boot/vmlinuz-4.18.0-330.rt7.111.el8.x86_64

2. Add a specific kdump kernel to the system’s Grand Unified Bootloader (GRUB) configuration.

   For example:

   # grubby --update-kernel=vmlinuz-4.18.0-330.el8.x86_64 --args="crashkernel=xxM"

   xxM is the required memory reserve in megabytes.

3. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. To stop the kdump service in the current session:

   # systemctl stop kdump.service

2. To disable the kdump service:

   # systemctl disable kdump.service

Procedure

1. To configure final_action, edit the /etc/kdump.conf file and add one of the following options:

   • final_action reboot
   • final_action halt
   • final_action poweroff

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure:

1. To configure an action to take if the dump fails, edit the /etc/kdump.conf file and specify one of the failure_action options:

   • failure_action reboot
   • failure_action halt
   • failure_action poweroff
   • failure_action shell
   • failure_action dump_to_rootfs

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure

1. Install the kexec-tools package.

2. Configure the default value for crashkernel.

   # kdumpctl reset-crashkernel --fadump=on --kernel=ALL

3. Optional: Reserve boot memory instead of the default value.

   # grubby --update-kernel ALL --args="fadump=on crashkernel=xxM"

   xxM is the required memory size in megabytes.

   Note

   When specifying boot configuration options, test the configurations by rebooting the kernel with kdump enabled. If the kdump kernel fails to boot, increase the crashkernel value gradually to set an appropriate value.

4. Reboot for changes to take effect.

   # reboot

Procedure

1. Add or edit the following lines in the /etc/sysctl.conf file to ensure that kdump starts as expected for sadump:

   kernel.panic=0
   kernel.unknown_nmi_panic=1

   Warning

   In particular, ensure that after kdump, the system does not reboot. If the system reboots after kdump has failed to save the vmcore file, then it is not possible to invoke the sadump.

2. Set the failure_action parameter in /etc/kdump.conf appropriately as halt or shell.

   failure_action shell

Procedure

1. Enable the relevant repositories:

   # subscription-manager repos --enable baseos repository

   # subscription-manager repos --enable appstream repository

   # subscription-manager repos --enable rhel-9-for-x86_64-baseos-debug-rpms

2. Install the crash package:

   # dnf install crash

3. Install the kernel-debuginfo package:

   # dnf install kernel-debuginfo

   The package kernel-debuginfo will correspond to the running kernel and provides the data necessary for the dump analysis.

Procedure

1. To start the crash utility, two necessary parameters need to be passed to the command:

   • The debug-info (a decompressed vmlinuz image), for example /usr/lib/debug/lib/modules/5.14.0-1.el9.x86_64/vmlinux provided through a specific kernel-debuginfo package.

   • The actual vmcore file, for example /var/crash/127.0.0.1-2021-09-13-14:05:33/vmcore

     The resulting crash command then looks like this:

     # crash /usr/lib/debug/lib/modules/5.14.0-1.el9.x86_64/vmlinux /var/crash/127.0.0.1-2021-09-13-14:05:33/vmcore

     Use the same <kernel> version that was captured by kdump.

2. Running the crash utility

   The following example shows analyzing a core dump created on September 13 2021 at 14:05 PM, using the 5.14.0-1.el9.x86_64 kernel.

   ...
   WARNING: kernel relocated [202MB]: patching 90160 gdb minimal_symbol values
   
         KERNEL: /usr/lib/debug/lib/modules/5.14.0-1.el9.x86_64/vmlinux
       DUMPFILE: /var/crash/127.0.0.1-2021-09-13-14:05:33/vmcore  [PARTIAL DUMP]
           CPUS: 2
           DATE: Mon Sep 13 14:05:16 2021
         UPTIME: 01:03:57
   LOAD AVERAGE: 0.00, 0.00, 0.00
          TASKS: 586
       NODENAME: localhost.localdomain
        RELEASE: 5.14.0-1.el9.x86_64
        VERSION: #1 SMP Wed Aug 29 11:51:55 UTC 2018
        MACHINE: x86_64  (2904 Mhz)
         MEMORY: 2.9 GB
          PANIC: "sysrq: SysRq : Trigger a crash"
            PID: 10635
        COMMAND: "bash"
           TASK: ffff8d6c84271800  [THREAD_INFO: ffff8d6c84271800]
            CPU: 1
          STATE: TASK_RUNNING (SYSRQ)
   
   crash>

3. To exit the interactive prompt and stop crash, type exit or q.

   crash> exit
   ~]#

Procedure

1. Access the Kernel Oops Analyzer tool.

2. To diagnose a kernel crash issue, upload a kernel oops log generated in vmcore.

   • Alternatively you can also diagnose a kernel crash issue by providing a text message or a vmcore-dmesg.txt as an input.+
3. Click DETECT to compare the oops message based on information from the makedumpfile against known solutions.

Procedure

1. Verify that the kdump service is enabled and active:

   # systemctl is-enabled kdump.service && systemctl is-active kdump.service
   enabled
   active

   If kdump is not enabled and running, set all required configurations and verify that kdump service is enabled.

2. Rebuild the initramfs image of the booting kernel with the early kdump functionality:

   # dracut -f --add earlykdump

3. Add the rd.earlykdump kernel command line parameter:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="rd.earlykdump"

4. Reboot the system to reflect the changes:

   # reboot

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Import your public key into the MOK list:

   # mokutil --import sb_cert.cer

3. Enter a new password for this MOK enrollment request.

4. Reboot the machine.

   The shim boot loader notices the pending MOK key enrollment request and it launches MokManager.efi to enable you to complete the enrollment from the UEFI console.

5. Choose Enroll MOK, enter the password you previously associated with this request when prompted, and confirm the enrollment.

   Your public key is added to the MOK list, which is persistent.

   Once a key is on the MOK list, it will be automatically propagated to the .platform keyring on this and subsequent boots when UEFI Secure Boot is enabled.

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Extract the key from the NSS database as a PKCS #12 file:

   # pk12util -o sb_cert.p12 \
              -n 'Custom Secure Boot key' \
              -d /etc/pki/pesign

3. When the previous command prompts you, enter a new password that encrypts the private key.

4. Export the unencrypted private key:

   # openssl pkcs12 \
            -in sb_cert.p12 \
            -out sb_cert.priv \
            -nocerts \
            -noenc

   Important

   Handle the unencrypted private key with care.

5. Sign your kernel module. The following command appends the signature directly to the ELF image in your kernel module file:

   # /usr/src/kernels/$(uname -r)/scripts/sign-file \
             sha256 \
             sb_cert.priv \
             sb_cert.cer \
             my_module.ko

Verification

1. Display information about the kernel module’s signature:

   # modinfo my_module.ko | grep signer
     signer:         Your Name Key

   Check that the signature lists your name as entered during generation.

   Note

   The appended signature is not contained in an ELF image section and is not a formal part of the ELF image. Therefore, utilities such as readelf cannot display the signature on your kernel module.

2. Load the module:

   # insmod my_module.ko

3. Remove (unload) the module:

   # modprobe -r my_module.ko

Procedure

1. Verify that your public keys are on the system keyring:

   # keyctl list %:.platform

2. Copy the kernel module into the extra/ directory of the kernel that you want:

   # cp my_module.ko /lib/modules/$(uname -r)/extra/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v my_module

5. Optional: To load the module on boot, add it to the /etc/modules-loaded.d/my_module.conf file:

   # echo "my_module" > /etc/modules-load.d/my_module.conf

Procedure

1. Determine the current version of the Revocation List:

   # fwupdmgr get-devices

   See the Current version field under UEFI dbx.

2. Enable the LVFS Revocation List repository:

   # fwupdmgr enable-remote lvfs

3. Refresh the repository metadata:

   # fwupdmgr refresh

4. Apply the Revocation List update:

   • On the command line:

     # fwupdmgr update

   • In the graphical interface:

     1. Open the Software application
     2. Navigate to the Updates tab.
     3. Find the Secure Boot dbx Configuration Update entry.
     4. Click Update.
5. At the end of the update, fwupdmgr or Software asks you to reboot the system. Confirm the reboot.

Procedure

1. Determine the current version of the Revocation List:

   # fwupdmgr get-devices

   See the Current version field under UEFI dbx.

2. List the updates available from RHEL:

   # ls /usr/share/dbxtool/

3. Select the most recent update file for your architecture. The file names use the following format:

   DBXUpdate-date-architecture.cab

4. Install the selected update file:

   # fwupdmgr install /usr/share/dbxtool/DBXUpdate-date-architecture.cab

5. At the end of the update, fwupdmgr asks you to reboot the system. Confirm the reboot.

Procedure

1. Create a 2048-bit RSA key with an SHA-256 primary storage key with a persistent handle of, for example, 81000001, by using one of the following utilities:

   1. By using the tss2 package:

      # TPM_DEVICE=/dev/tpm0 tsscreateprimary -hi o -st
      Handle 80000000
      # TPM_DEVICE=/dev/tpm0 tssevictcontrol -hi o -ho 80000000 -hp 81000001

   2. By using the tpm2-tools package:

      # tpm2_createprimary --key-algorithm=rsa2048 --key-context=key.ctxt
      name-alg:
        value: sha256
        raw: 0xb
      …
      sym-keybits: 128
      rsa: xxxxxx…
      
      # tpm2_evictcontrol -c key.ctxt 0x81000001
      persistentHandle: 0x81000001
      action: persisted

2. Create a trusted key by using a TPM 2.0 with the syntax of keyctl add trusted <NAME> "new <KEY_LENGTH> keyhandle=<PERSISTENT-HANDLE> [options]" <KEYRING>. In this example, the persistent handle is 81000001.

   # keyctl add trusted kmk "new 32 keyhandle=0x81000001" @u
   642500861

   The command creates a trusted key called kmk with the length of 32 bytes (256 bits) and places it in the user keyring (@u). The keys may have a length of 32 to 128 bytes (256 to 1024 bits).

3. List the current structure of the kernel keyrings:

   # keyctl show
   Session Keyring
          -3 --alswrv    500   500  keyring: ses 97833714 --alswrv 500 -1 \ keyring: uid.1000 642500861 --alswrv 500 500 \ trusted: kmk

4. Export the key to a user-space blob by using the serial number of the trusted key:

   # keyctl pipe 642500861 > kmk.blob

   The command uses the pipe subcommand and the serial number of kmk.

5. Load the trusted key from the user-space blob:

   # keyctl add trusted kmk "load `cat kmk.blob`" @u
   268728824

6. Create secure encrypted keys that use the TPM-sealed trusted key (kmk). Follow this syntax: keyctl add encrypted <NAME> "new [FORMAT] <KEY_TYPE>:<PRIMARY_KEY_NAME> <KEY_LENGTH>" <KEYRING>:

   # keyctl add encrypted encr-key "new trusted:kmk 32" @u
   159771175

Procedure

1. Generate a user key by using a random sequence of numbers.

   # keyctl add user kmk-user "$(dd if=/dev/urandom bs=1 count=32 2>/dev/null)" @u
   427069434

   The command generates a user key called kmk-user which acts as a primary key and is used to seal the actual encrypted keys.

2. Generate an encrypted key using the primary key from the previous step:

   # keyctl add encrypted encr-key "new user:kmk-user 32" @u
   1012412758

Verification

1. List all keys in the specified user keyring:

   # keyctl list @u
   2 keys in keyring:
   427069434: --alswrv  1000  1000 user: kmk-user
   1012412758: --alswrv  1000  1000 encrypted: encr-key

Procedure

1. Enable IMA and EVM in the fix mode for the current boot entry and allow users to gather and update the IMA measurements by adding the following kernel command-line parameters:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="ima_policy=appraise_tcb ima_appraise=fix evm=fix"

   The command enables IMA and EVM in the fix mode for the current boot entry and allows users to gather and update the IMA measurements.

   The ima_policy=appraise_tcb kernel command-line parameter ensures that the kernel uses the default Trusted Computing Base (TCB) measurement policy and the appraisal step. The appraisal step forbids access to files whose prior and current measures do not match.

2. Reboot to make the changes come into effect.

3. Optional: Verify that the parameters have been added to the kernel command line:

   # cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-1.el9.x86_64 root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet ima_policy=appraise_tcb ima_appraise=fix evm=fix

4. Create a kernel master key to protect the EVM key:

   # keyctl add user kmk "$(dd if=/dev/urandom bs=1 count=32 2> /dev/null)" @u
   748544121

   The kmk is kept entirely in the kernel space memory. The 32-byte long value of the kmk is generated from random bytes from the /dev/urandom file and placed in the user (@u) keyring. The key serial number is on the first line of the previous output.

5. Create an encrypted EVM key based on the kmk:

   # keyctl add encrypted evm-key "new user:kmk 64" @u
   641780271

   The command uses the kmk to generate and encrypt a 64-byte long user key (named evm-key) and places it in the user (@u) keyring. The key serial number is on the first line of the previous output.

   Important

   It is necessary to name the user key as evm-key because that is the name the EVM subsystem is expecting and is working with.

6. Create a directory for exported keys.

   # mkdir -p /etc/keys/

7. Search for the kmk and export its unencrypted value into the new directory.

   # keyctl pipe $(keyctl search @u user kmk) > /etc/keys/kmk

8. Search for the evm-key and export its encrypted value into the new directory.

   # keyctl pipe $(keyctl search @u encrypted evm-key) > /etc/keys/evm-key

   The evm-key has been encrypted by the kernel master key earlier.

9. Optional: View the newly created keys.

   # keyctl show
   Session Keyring
   974575405   --alswrv     0        0      keyring: ses 299489774 --alswrv 0 65534 \ keyring: uid.0 748544121 --alswrv 0 0 \ user: kmk
   641780271   --alswrv     0        0           \_ encrypted: evm-key
   
   # ls -l /etc/keys/
   total 8
   -rw-r--r--. 1 root root 246 Jun 24 12:44 evm-key
   -rw-r--r--. 1 root root  32 Jun 24 12:43 kmk

10. Optional: If the keys have been removed from the keyring, for example after system reboot, you can import the already exported kmk and evm-key instead of creating new ones.

    1. Import the kmk.

       # keyctl add user kmk "$(cat /etc/keys/kmk)" @u
       451342217

    2. Import the evm-key.

       # keyctl add encrypted evm-key "load $(cat /etc/keys/evm-key)" @u
       924537557

11. Activate EVM.

    # echo 1 > /sys/kernel/security/evm

12. Relabel the whole system.

    # find / -fstype xfs -type f -uid 0 -exec head -n 1 '{}' >/dev/null \;

    Warning

    Enabling IMA and EVM without relabeling the system might make the majority of the files on the system inaccessible.

Procedure

1. Create a test file:

   # echo <Test_text> > test_file

   IMA and EVM ensure that the test_file example file has assigned hash values that are stored as its extended attributes.

2. Inspect the file’s extended attributes:

   # getfattr -m . -d test_file
   # file: test_file
   security.evm=0sAnDIy4VPA0HArpPO/EqiutnNyBql
   security.ima=0sAQOEDeuUnWzwwKYk+n66h/vby3eD

   The example output shows extended attributes with the IMA and EVM hash values and SELinux context. EVM adds a security.evm extended attribute related to the other attributes. At this point, you can use the evmctl utility on security.evm to generate either an RSA-based digital signature or a Hash-based Message Authentication Code (HMAC-SHA1).

Procedure

1. Install the rpm-plugin-ima plug-in:

   # dnf install rpm-plugin-ima -y

2. Reinstall all packages:

   # dnf reinstall '*' -y

Verification

1. Confirm that the reinstalled package file has a valid IMA signature. For example, to check the IMA signature of the /usr/bin/bash file, run the following command:

   # getfattr -m security.ima -d /usr/bin/bash

   'security.ima=0sAwIE0zIESQBnMGUCMFhf0iBeM7NjjhCCHVt4/ORx1eCegjrWSHzFbJMCsAhR9bYU2hNGjiWUYT2IIqWaaAIxALFGUkqGP5vDLuxQXibO9g7HFcfyZzRBY4rbKPsXcAIZRtDHVS5dQBZqM3hyS5v1MA=='

2. Verify the IMA signature of a file with a specified certificate. For example, to check that the IMA signature of /usr/bin/bash is accessible by /usr/share/doc/kernel-keys/$(uname -r)/ima.cer, run the following command:".

   # evmctl ima_verify -k /usr/share/doc/kernel-keys/$(uname -r)/ima.cer /usr/bin/bash

   'key 1: d3320449 /usr/share/doc/kernel-keys/5.14.0-359.el9.x86-64/ima.cer
   /usr/bin/bash:' verification is OK

Procedure

1. To copy the Red Hat IMA code signing key to the /etc/ima/keys file, run:

   $ mkdir -p /etc/keys/ima
   $ cp /usr/share/doc/kernel-keys/$(uname -r)/ima.cer /etc/ima/keys

2. To add the IMA code signing key to the .ima keyring, run:

   # keyctl padd asymmetric RedHat-IMA %:.ima < /etc/ima/keys/ima.cer

3. Depending on your threat model, define an IMA policy in the /etc/sysconfig/ima-policy file. For example, the following IMA policy checks the integrity of both executables and involved memory mapping library files:

   # PROC_SUPER_MAGIC = 0x9fa0
   dont_appraise fsmagic=0x9fa0
   # SYSFS_MAGIC = 0x62656572
   dont_appraise fsmagic=0x62656572
   # DEBUGFS_MAGIC = 0x64626720
   dont_appraise fsmagic=0x64626720
   # TMPFS_MAGIC = 0x01021994
   dont_appraise fsmagic=0x1021994
   # RAMFS_MAGIC
   dont_appraise fsmagic=0x858458f6
   # DEVPTS_SUPER_MAGIC=0x1cd1
   dont_appraise fsmagic=0x1cd1
   # BINFMTFS_MAGIC=0x42494e4d
   dont_appraise fsmagic=0x42494e4d
   # SECURITYFS_MAGIC=0x73636673
   dont_appraise fsmagic=0x73636673
   # SELINUX_MAGIC=0xf97cff8c
   dont_appraise fsmagic=0xf97cff8c
   # SMACK_MAGIC=0x43415d53
   dont_appraise fsmagic=0x43415d53
   # NSFS_MAGIC=0x6e736673
   dont_appraise fsmagic=0x6e736673
   # EFIVARFS_MAGIC
   dont_appraise fsmagic=0xde5e81e4
   # CGROUP_SUPER_MAGIC=0x27e0eb
   dont_appraise fsmagic=0x27e0eb
   # CGROUP2_SUPER_MAGIC=0x63677270
   dont_appraise fsmagic=0x63677270
   appraise func=BPRM_CHECK
   appraise func=FILE_MMAP mask=MAY_EXEC

4. To load the IMA policy to make sure the kernel accepts this IMA policy, run:

   # echo /etc/sysconfig/ima-policy > /sys/kernel/security/ima/policy
   # echo $?
   0

5. To enable the dracut integrity module to automatically load the IMA code signing key and the IMA policy, run:

   # echo 'add_dracutmodules+=" integrity "' > /etc/dracut.conf.d/98-integrity.conf
   # dracut -f

Procedure

1. To generate a custom IMA CA key pair, run:

   # openssl req -new -x509 -utf8 -sha256 -days 3650 -batch -config ima_ca.conf -outform DER -out custom_ima_ca.der -keyout custom_ima_ca.priv

2. Optional: To check the content of the ima_ca.conf file, run:

   # cat ima_ca.conf
   [ req ]
   default_bits = 2048
   distinguished_name = req_distinguished_name
   prompt = no
   string_mask = utf8only
   x509_extensions = ca
   
   [ req_distinguished_name ]
   O = YOUR_ORG
   CN =  YOUR_COMMON_NAME IMA CA
   emailAddress = YOUR_EMAIL
   
   [ ca ]
   basicConstraints=critical,CA:TRUE
   subjectKeyIdentifier=hash
   authorityKeyIdentifier=keyid:always,issuer
   keyUsage=critical,keyCertSign,cRLSign

3. To generate a private key and a certificate signing request (CSR) for the IMA code signing key, run:

   # openssl req -new -utf8 -sha256 -days 365 -batch -config ima.conf -out custom_ima.csr -keyout custom_ima.priv

4. Optional: To check the content of the ima.conf file, run:

   # cat ima.conf
   [ req ]
   default_bits = 2048
   distinguished_name = req_distinguished_name
   prompt = no
   string_mask = utf8only
   x509_extensions = code_signing
   
   [ req_distinguished_name ]
   O = YOUR_ORG
   CN = YOUR_COMMON_NAME IMA signing key
   emailAddress = YOUR_EMAIL
   
   [ code_signing ]
   basicConstraints=critical,CA:FALSE
   keyUsage=digitalSignature
   subjectKeyIdentifier=hash
   authorityKeyIdentifier=keyid:always,issuer

5. Use the IMA CA private key to sign the CSR to create the IMA code signing certificate:

   # openssl x509 -req -in custom_ima.csr -days 365 -extfile ima.conf -extensions code_signing -CA custom_ima_ca.der -CAkey custom_ima_ca.priv -CAcreateserial -outform DER -out ima.der

Procedure

1. Enable Secure Boot.

2. Permanently add the ima_policy=secure_boot kernel parameter.

   For instructions, see Configuring kernel parameters permanently with sysctl.

3. Prepare your IMA policy by running the command:

   # evmctl ima_sign /etc/sysconfig/ima-policy -k <PATH_TO_YOUR_CUSTOM_IMA_KEY>
   Place your public certificate under /etc/keys/ima/ and add it to the .ima keyring

4. Sign the policy with your custom IMA code signing key by running the command:

   # keyctl padd asymmetric CUSTOM_IMA1 %:.ima < /etc/ima/keys/my_ima.cer

5. Load the IMA policy by running the command:

   # echo /etc/sysconfig/ima-policy > /sys/kernel/security/ima/policy
   # echo $?
   0

1. Check the assigned values for the service of your choice.

   # systemctl show --property <unit file option> <service name>

2. Set the required value of the CPU time allocation policy option:

   # systemctl set-property <service name> <unit file option>=<value>

Procedure

1. To view which cgroup a process belongs to, run the # cat proc/<PID>/cgroup command:

   # cat /proc/2467/cgroup
   0::/system.slice/example.service

   The example output relates to a process of interest. In this case, it is a process identified by PID 2467, which belongs to the example.service unit. You can determine whether the process was placed in a correct control group as defined by the systemd unit file specifications.

2. To display what controllers and respective configuration files the cgroup uses, check the cgroup directory:

   # cat /sys/fs/cgroup/system.slice/example.service/cgroup.controllers
   memory pids
   
   # ls /sys/fs/cgroup/system.slice/example.service/
   cgroup.controllers
   cgroup.events
   …​
   cpu.pressure
   cpu.stat
   io.pressure
   memory.current
   memory.events
   …​
   pids.current
   pids.events
   pids.max

Procedure

1. Display a dynamic account of currently running cgroups with the systemd-cgtop command.

   # systemd-cgtop
   Control Group                            Tasks   %CPU   Memory  Input/s Output/s
   /                                          607   29.8     1.5G        -        -
   /system.slice                              125      -   428.7M        -        -
   /system.slice/ModemManager.service           3      -     8.6M        -        -
   /system.slice/NetworkManager.service         3      -    12.8M        -        -
   /system.slice/accounts-daemon.service        3      -     1.8M        -        -
   /system.slice/boot.mount                     -      -    48.0K        -        -
   /system.slice/chronyd.service                1      -     2.0M        -        -
   /system.slice/cockpit.socket                 -      -     1.3M        -        -
   /system.slice/colord.service                 3      -     3.5M        -        -
   /system.slice/crond.service                  1      -     1.8M        -        -
   /system.slice/cups.service                   1      -     3.1M        -        -
   /system.slice/dev-hugepages.mount            -      -   244.0K        -        -
   /system.slice/dev-mapper-rhel\x2dswap.swap   -      -   912.0K        -        -
   /system.slice/dev-mqueue.mount               -      -    48.0K        -        -
   /system.slice/example.service                2      -     2.0M        -        -
   /system.slice/firewalld.service              2      -    28.8M        -        -
   ...

   The example output displays currently running cgroups ordered by their resource usage (CPU, memory, disk I/O load). The list refreshes every 1 second by default. Therefore, it offers a dynamic insight into the actual resource usage of each control group.

Procedure

1. Edit the /usr/lib/systemd/system/example.service file to limit the memory usage of a service:

   …​
   [Service]
   MemoryMax=1500K
   …​

   The configuration limits the maximum memory that the processes in a control group cannot exceed. The example.service service is part of such a control group which has imposed limitations. You can use suffixes K, M, G, or T to identify Kilobyte, Megabyte, Gigabyte, or Terabyte as a unit of measurement.

2. Reload all unit configuration files:

   # systemctl daemon-reload

3. Restart the service:

   # systemctl restart example.service

Verification

1. Check that the changes took effect:

   # cat /sys/fs/cgroup/system.slice/example.service/memory.max
   1536000

   The example output shows that the memory consumption was limited at around 1,500 KB.

1. Check the values of the CPUAffinity unit file option in the service of your choice:

   $ systemctl show --property <CPU affinity configuration option> <service name>

2. As the root user, set the required value of the CPUAffinity unit file option for the CPU ranges used as the affinity mask:

   # systemctl set-property <service name> CPUAffinity=<value>

3. Restart the service to apply the changes.

   # systemctl restart <service name>