Managing, monitoring, and updating the kernel

Procedure

1. List the available versions of a package:

   $ yum repoquery <package_name>

   For example, list the available versions of the kernel-core package:

   $ yum repoquery kernel-core
   kernel-core-0:4.18.0-147.0.2.el8_1.x86_64
   kernel-core-0:4.18.0-147.0.3.el8_1.x86_64
   kernel-core-0:4.18.0-147.3.1.el8_1.x86_64
   kernel-core-0:4.18.0-147.5.1.el8_1.x86_64
   kernel-core-0:4.18.0-147.8.1.el8_1.x86_64
   kernel-core-0:4.18.0-147.el8.x86_64
   …​

2. Display the list of files in a package:

   $ yum repoquery -l <package_name>

   For example, display the list of files in the kernel-core-0:5.14.0-162.23.1.el9_1.x86_64 package.

   $ yum repoquery -l kernel-core-0:4.18.0-147.8.1.el8_1.x86_64
   /boot/.vmlinuz-4.18.0-147.8.1.el8_1.x86_64.hmac
   /boot/System.map-4.18.0-147.8.1.el8_1.x86_64
   /boot/config-4.18.0-147.8.1.el8_1.x86_64
   /boot/initramfs-4.18.0-147.8.1.el8_1.x86_64.img
   /boot/symvers-4.18.0-147.8.1.el8_1.x86_64.gz
   /boot/vmlinuz-4.18.0-147.8.1.el8_1.x86_64
   /etc/ld.so.conf.d/kernel-4.18.0-147.8.1.el8_1.x86_64.conf
   /lib/modules
   /lib/modules/4.18.0-147.8.1.el8_1.x86_64
   ...

Procedure

1. To update the kernel, enter the following command:

   # yum update kernel

   This command updates the kernel along with all dependencies to the latest available version.

2. Reboot your system for the changes to take effect.

Procedure

1. Select a kernel module you want to load.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Load the relevant kernel module:

   # modprobe <MODULE_NAME>

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. List all loaded kernel modules:

   # lsmod

2. Select the kernel module you want to unload.

   If a kernel module has dependencies, unload those prior to unloading the kernel module. For details on identifying modules with dependencies, see Listing currently loaded kernel modules and Kernel module dependencies.

3. Unload the relevant kernel module:

   # modprobe -r <MODULE_NAME>

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

Procedure

1. Boot the system into the boot loader.
2. Use the cursor keys to highlight the relevant boot loader entry.

3. Press the e key to edit the entry.

   Figure 2.1. Kernel boot menu

   
4. Use the cursor keys to navigate to the line that starts with linux.

5. Append modprobe.blacklist=module_name to the end of the line.

   Figure 2.2. Kernel boot entry

   

   The serio_raw kernel module illustrates a rogue module to be unloaded early in the boot process.

6. Press Ctrl+X to boot using the modified configuration.

Procedure

1. Select a kernel module you want to load during the boot process.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for the module:

   # echo <MODULE_NAME> > /etc/modules-load.d/<MODULE_NAME>.conf

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Optionally, after reboot, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   The example command above should succeed and display the relevant kernel module.

Procedure

1. List modules loaded to the currently running kernel by using the lsmod command:

   $ lsmod
   Module                  Size  Used by
   tls                   131072  0
   uinput                 20480  1
   snd_seq_dummy          16384  0
   snd_hrtimer            16384  1
   …

   In the output, identify the module you want to prevent from being loaded.

   • Alternatively, identify an unloaded kernel module you want to prevent from potentially loading in the /lib/modules/<KERNEL-VERSION>/kernel/<SUBSYSTEM>/ directory, for example:

     $ ls /lib/modules/4.18.0-477.20.1.el8_8.x86_64/kernel/crypto/
     ansi_cprng.ko.xz        chacha20poly1305.ko.xz  md4.ko.xz               serpent_generic.ko.xz
     anubis.ko.xz            cmac.ko.xz…

2. Create a configuration file serving as a denylist:

   # touch /etc/modprobe.d/denylist.conf

3. In a text editor of your choice, combine the names of modules you want to exclude from automatic loading to the kernel with the blacklist configuration command, for example:

   # Prevents <KERNEL-MODULE-1> from being loaded
   blacklist <MODULE-NAME-1>
   install <MODULE-NAME-1> /bin/false
   
   # Prevents <KERNEL-MODULE-2> from being loaded
   blacklist <MODULE-NAME-2>
   install <MODULE-NAME-2> /bin/false
   …

   Because the blacklist command does not prevent the module from being loaded as a dependency for another kernel module that is not in a denylist, you must also define the install line. In this case, the system runs /bin/false instead of installing the module. The lines starting with a hash sign are comments you can use to make the file more readable.

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

4. Create a backup copy of the current initial RAM disk image before rebuilding:

   # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

   • Alternatively, create a backup copy of an initial RAM disk image which corresponds to the kernel version for which you want to prevent kernel modules from automatic loading:

     # cp /boot/initramfs-<VERSION>.img /boot/initramfs-<VERSION>.img.bak.$(date +%m-%d-%H%M%S)

5. Generate a new initial RAM disk image to apply the changes:

   # dracut -f -v

   • If you build an initial RAM disk image for a different kernel version than your system currently uses, specify both target initramfs and kernel version:

     # dracut -f -v /boot/initramfs-<TARGET-VERSION>.img <CORRESPONDING-TARGET-KERNEL-VERSION>

6. Restart the system:

   $ reboot

Procedure

1. Create the /root/testmodule/test.c file with the following content.

   #include <linux/module.h>
   #include <linux/kernel.h>
   
   int init_module(void)
       { printk("Hello World\n This is a test\n"); return 0; }
   
   void cleanup_module(void)
       { printk("Good Bye World"); }

   The test.c file is a source file that provides the main functionality to the kernel module. The file has been created in a dedicated /root/testmodule/ directory for organizational purposes. After the module compilation, the /root/testmodule/ directory will contain multiple files.

   The test.c file includes from the system libraries:

   • The linux/kernel.h header file is necessary for the printk() function in the example code.
   • The linux/module.h file contains function declarations and macro definitions to be shared between several source files written in C programming language.
2. Follow the init_module() and cleanup_module() functions to start and end the kernel logging function printk(), which prints text.

3. Create the /root/testmodule/Makefile file with the following content.

   obj-m := test.o

   The Makefile contains instructions that the compiler has to produce an object file specifically named test.o. The obj-m directive specifies that the resulting test.ko file is going to be compiled as a loadable kernel module. Alternatively, the obj-y directive would instruct to build test.ko as a built-in kernel module.

4. Compile the kernel module.

   # make -C /lib/modules/$(uname -r)/build M=/root/testmodule modules
   make: Entering directory '/usr/src/kernels/4.18.0-305.el8.x86_64'
     CC [M]  /root/testmodule/test.o
     Building modules, stage 2.
     MODPOST 1 modules
   WARNING: modpost: missing MODULE_LICENSE() in /root/testmodule/test.o
   see include/linux/module.h for more information
     CC      /root/testmodule/test.mod.o
     LD [M]  /root/testmodule/test.ko
   make: Leaving directory '/usr/src/kernels/4.18.0-305.el8.x86_64'

   The compiler creates an object file (test.o) for each source file (test.c) as an intermediate step before linking them together into the final kernel module (test.ko).

   After a successful compilation, /root/testmodule/ contains additional files that relate to the compiled custom kernel module. The compiled module itself is represented by the test.ko file.

Verification

1. Optional: check the contents of the /root/testmodule/ directory:

   # ls -l /root/testmodule/
   total 152
   -rw-r—​r--. 1 root root    16 Jul 26 08:19 Makefile
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 modules.order
   -rw-r—​r--. 1 root root     0 Jul 26 08:20 Module.symvers
   -rw-r—​r--. 1 root root   224 Jul 26 08:18 test.c
   -rw-r—​r--. 1 root root 62176 Jul 26 08:20 test.ko
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 test.mod
   -rw-r—​r--. 1 root root   849 Jul 26 08:20 test.mod.c
   -rw-r—​r--. 1 root root 50936 Jul 26 08:20 test.mod.o
   -rw-r—​r--. 1 root root 12912 Jul 26 08:20 test.o

2. Copy the kernel module to the /lib/modules/$(uname -r)/ directory:

   # cp /root/testmodule/test.ko /lib/modules/$(uname -r)/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v test
   insmod /lib/modules/4.18.0-305.el8.x86_64/test.ko

5. Verify that the kernel module was successfully loaded:

   # lsmod | grep test
   test                   16384  0

6. Read the latest messages from the kernel ring buffer:

   # dmesg
   [74422.545004] Hello World
                   This is a test

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Import your public key into the MOK list:

   # mokutil --import sb_cert.cer

3. Enter a new password for this MOK enrollment request.

4. Reboot the machine.

   The shim boot loader notices the pending MOK key enrollment request and it launches MokManager.efi to enable you to complete the enrollment from the UEFI console.

5. Choose Enroll MOK, enter the password you previously associated with this request when prompted, and confirm the enrollment.

   Your public key is added to the MOK list, which is persistent.

   Once a key is on the MOK list, it will be automatically propagated to the .platform keyring on this and subsequent boots when UEFI Secure Boot is enabled.

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Extract the key from the NSS database as a PKCS #12 file:

   # pk12util -o sb_cert.p12 \
              -n 'Custom Secure Boot key' \
              -d /etc/pki/pesign

3. When the previous command prompts you, enter a new password that encrypts the private key.

4. Export the unencrypted private key:

   # openssl pkcs12 \
            -in sb_cert.p12 \
            -out sb_cert.priv \
            -nocerts \
            -nodes

   Important

   Handle the unencrypted private key with care.

5. Sign your kernel module. The following command appends the signature directly to the ELF image in your kernel module file:

   # /usr/src/kernels/$(uname -r)/scripts/sign-file \
             sha256 \
             sb_cert.priv \
             sb_cert.cer \
             my_module.ko

Verification

1. Display information about the kernel module’s signature:

   # modinfo my_module.ko | grep signer
     signer:         Your Name Key

   Check that the signature lists your name as entered during generation.

   Note

   The appended signature is not contained in an ELF image section and is not a formal part of the ELF image. Therefore, utilities such as readelf cannot display the signature on your kernel module.

2. Load the module:

   # insmod my_module.ko

3. Remove (unload) the module:

   # modprobe -r my_module.ko

Procedure

1. Verify that your public keys are on the system keyring:

   # keyctl list %:.platform

2. Copy the kernel module into the extra/ directory of the kernel that you want:

   # cp my_module.ko /lib/modules/$(uname -r)/extra/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v my_module

5. Optionally, to load the module on boot, add it to the /etc/modules-loaded.d/my_module.conf file:

   # echo "my_module" > /etc/modules-load.d/my_module.conf

Procedure

1. Boot into the GRUB 2 boot menu.
2. Select the kernel you want to start.
3. Press the e key to edit the kernel parameters.
4. Find the kernel command line by moving the cursor down. The kernel command line starts with linux on 64-Bit IBM Power Series and x86-64 BIOS-based systems, or linuxefi on UEFI systems.

5. Move the cursor to the end of the line.

   Note

   Press Ctrl+a to jump to the start of the line and Ctrl+e to jump to the end of the line. On some systems, Home and End keys might also work.

6. Edit the kernel parameters as required. For example, to run the system in emergency mode, add the emergency parameter at the end of the linux line:

   linux   ($root)/vmlinuz-4.18.0-348.12.2.el8_5.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet emergency

   To enable the system messages, remove the rhgb and quiet parameters.

7. Press Ctrl+x to boot with the selected kernel and the modified command line parameters.

Procedure

1. Add the following two lines to the /etc/default/grub file:

   GRUB_TERMINAL="serial"
   GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

   The first line disables the graphical terminal. The GRUB_TERMINAL key overrides values of GRUB_TERMINAL_INPUT and GRUB_TERMINAL_OUTPUT keys.

   The second line adjusts the baud rate (--speed), parity and other values to fit your environment and hardware. Note that a much higher baud rate, for example 115200, is preferable for tasks such as following log files.

2. Update the GRUB configuration file.

   • On BIOS-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines:

     # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

3. Reboot the system for the changes to take effect.

Procedure

1. List all parameters and their values.

   # sysctl -a

   Note

   The # sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

2. To configure a parameter temporarily, enter:

   # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

   The sample command above changes the parameter value while the system is running. The changes take effect immediately, without a need for restart.

   Note

   The changes return back to default after your system reboots.

Procedure

1. List all parameters.

   # sysctl -a

   The command displays all kernel parameters that can be configured at runtime.

2. Configure a parameter permanently:

   # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

   The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

Procedure

1. Create a new configuration file in /etc/sysctl.d/.

   # vim /etc/sysctl.d/<some_file.conf>

2. Include kernel parameters, one per line.

   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

3. Save the configuration file.

4. Reboot the machine for the changes to take effect.

   • Alternatively, to apply changes without rebooting, enter:

     # sysctl -p /etc/sysctl.d/<some_file.conf>

     The command enables you to read values from the configuration file, which you created earlier.

Procedure

1. Identify a kernel parameter you want to configure.

   # ls -l /proc/sys/<TUNABLE_CLASS>/

   The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

2. Assign a target value to the kernel parameter.

   # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

   The command makes configuration changes that will disappear once the system is restarted.

3. Optionally, verify the value of the newly set kernel parameter.

   # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

Procedure

1. On the GRUB boot screen, press the e key for edit.

2. Add the following parameter at the end of the linux line:

   systemd.unit=rescue.target

   

3. Press Ctrl+x to boot to rescue mode.

   

Procedure

1. On the GRUB boot screen, press the e key for edit.

2. Add the following parameter at the end of the linux line:

   systemd.unit=emergency.target

   

3. Press Ctrl+x to boot to emergency mode.

   

Procedure

1. On the GRUB boot screen, press the e key for edit.

2. Add the following parameter at the end of the linux line:

   systemd.debug-shell

   

3. Optionally add the debug option.

   Note

   Adding the debug option to the kernel command line increases the number of log messages. For systemd, the kernel command-line option debug is now a shortcut for systemd.log_level=debug.

4. Press Ctrl+x to boot to the debug shell.

Procedure

1. Press Ctrl+Alt+F9 to connect to the debug shell.

   If you work with a virtual machine, sending this key combination requires support from the virtualization application. For example, if you use Virtual Machine Manager, select Send Key → Ctrl+Alt+F9 from the menu.

2. The debug shell does not require authentication, therefore you can see a prompt similar to the following on TTY9:

Procedure

1. Boot the host from an installation source.

2. In the boot menu for the installation media, select the Troubleshooting option.

   

3. In the Troubleshooting menu, select the Rescue a Red Hat Enterprise Linux system option.

   

4. At the Rescue menu, select 1 and press the Enter key to continue.

   

5. Change the file system root as follows:

   sh-4.4# chroot /mnt/sysimage

   

6. Enter the passwd command and follow the instructions displayed on the command line to change the root password.

   

7. Remove the autorelable file to prevent a time consuming SELinux relabel of the disk:

   sh-4.4# rm -f /.autorelabel

8. Enter the exit command to exit the chroot environment.
9. Enter the exit command again to resume the initialization and finish the system boot.

Procedure

1. Start the system and, on the GRUB boot screen, press the e key for edit.

2. Add the rd.break parameter at the end of the linux line:

   

3. Press Ctrl+x to boot the system with the changed parameters.

   

4. Remount the file system as writable.

   switch_root:/# mount -o remount,rw /sysroot

5. Change the file system’s root.

   switch_root:/# chroot /sysroot

6. Enter the passwd command and follow the instructions displayed on the command line.

   

7. Relabel all files on the next system boot.

   sh-4.4# touch /.autorelabel

8. Remount the file system as read only:

   sh-4.4# mount -o remount,ro /

9. Enter the exit command to exit the chroot environment.

10. Enter the exit command again to resume the initialization and finish the system boot.

    Note

    The SELinux relabeling process can take a long time. A system reboot occurs automatically when the process is complete.

Procedure

1. Copy all the kernel arguments from your default kernel to this new kernel entry.

   # grubby --add-kernel=new_kernel --title="entry_title" --initrd="new_initrd" --copy-default

2. Get the list of available boot entries.

   # ls -l /boot/loader/entries/*
   -rw-r--r--. 1 root root 408 May 27 06:18 /boot/loader/entries/67db13ba8cdb420794ef3ee0a8313205-0-rescue.conf
   -rw-r--r--. 1 root root 536 Jun 30 07:53 /boot/loader/entries/67db13ba8cdb420794ef3ee0a8313205-4.18.0-372.9.1.el8.x86_64.conf
   -rw-r--r--  1 root root 336 Aug 15 15:12 /boot/loader/entries/d88fa2c7ff574ae782ec8c4288de4e85-4.18.0-193.el8.x86_64.conf

3. Create a new boot entry. For example, for the 4.18.0-193.el8.x86_64 kernel, issue the command as follows:

   # grubby --grub2 --add-kernel=/boot/vmlinuz-4.18.0-193.el8.x86_64 --title="Red Hat Enterprise 8 Test" --initrd=/boot/initramfs-4.18.0-193.el8.x86_64.img --copy-default

Procedure

1. List the kernel parameters from the kernelopts variable:

   # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet

2. Make the changes to the kernel command-line parameters. You can add, remove or modify a parameter. For example, to add the debug parameter, enter:

   # grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) <debug>"

3. Optional: verify that the newly added parameter has been added to kernelopts:

   # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet debug

4. Reboot the system for the changes to take effect.

Procedure

1. Set the GRUB_TIMEOUT_STYLE option in the /etc/default/grub file as follows:

   GRUB_TIMEOUT_STYLE=hidden

2. Rebuild the grub.cfg file for the changes to take effect.

   • On BIOS-based machines, enter:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines, enter:

     # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

3. Press the Esc key to display the list of bootable kernels when booting.

Procedure

1. Specify which operating system or kernel should be loaded by default by passing its index to the grub2-set-default command. For example:

   # grubby --set-default-index=1
   The default is /boot/loader/entries/d5151aa93c444ac89e78347a1504d6c6-4.18.0-348.el8.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-348.el8.x86_64

   GRUB supports using a numeric value as the key for the saved_entry directive in /boot/grub2/grubenv to change the default order in which the operating systems are loaded.

   Note

   Index counting starts with zero; therefore, in the previous example, GRUB loads the second entry. With the next installed kernel, the index value will be overwritten.

   Note

   You can also use grubby to find indices for kernels. For more information, see Viewing the GRUB Menu Entry for a Kernel.

2. Optional: Force the system to always use a particular menu entry:

   1. List the available menu entries:

      # grubby --info=ALL

   2. Use the menu entry name or the number of the position of a menu entry in the list as the key to the GRUB_DEFAULT directive in the /etc/default/grub file. For example:

      GRUB_DEFAULT=example-gnu-linux

3. Rebuild the grub.cfg file for the changes to take effect.

   • On BIOS-based machines, enter:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines:

     # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Procedure

1. Reinstall GRUB on the device where it is installed. For example, if sda is your device:

   # grub2-install /dev/sda

2. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Reinstall the grub2-efi and shim boot loader files:

   # yum reinstall grub2-efi shim

2. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Determine the disk partition that stores GRUB:

   # bootlist -m normal -o
   sda1

2. Reinstall GRUB on the disk partition:

   # grub2-install partition

   Replace partition with the GRUB partition that you found in the previous step, such as /dev/sda1.

3. Reboot your system for the changes to take effect:

   # reboot

Procedure

1. Remove the configuration files.

   # rm /etc/grub.d/*
   # rm /etc/sysconfig/grub

2. Reinstall packages.

   • On BIOS-based machines, enter:

     # yum reinstall grub2-tools

   • On UEFI-based machines, enter:

     # yum reinstall grub2-efi shim grub2-tools

3. Rebuild the grub.cfg file for the changes to take effect.

   • On BIOS-based machines, enter:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines, enter:

     # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

4. Follow Reinstalling GRUB procedure to restore GRUB on the /boot/ partition.

Procedure

1. Issue the grub2-setpassword command as root.

   # grub2-setpassword

2. Enter the password for the user and press the Enter key to confirm the password.

   Enter password:
   Confirm the password:

Procedure

1. Open the Boot Loader Specification (BLS) file for boot entry you want to modify from the /boot/loader/entries/ directory.
2. Find the line beginning with grub_users. This parameter passes extra arguments to menuentry.

3. Set the grub_users attribute to the user name that is allowed to boot the entry besides the superusers, by default this user is root. Here is a sample configuration file:

   title Red Hat Enterprise Linux (4.18.0-221.el8.x86_64) 8.3
   (Ootpa)
   version 4.18.0-221.el8.x86_64
   linux /vmlinuz-4.18.0-221.el8.x86_64
   initrd /initramfs-4.18.0-221.el8.x86_64.img $tuned_initrd
   options $kernelopts $tuned_params
   id rhel-20200625210904-4.18.0-221.el8.x86_64
   grub_users root
   grub_arg --unrestricted
   grub_class kernel

4. Save and close the BLS file.

1. Console log-level, defines the lowest priority of messages printed to the console.
2. Default log-level for messages without an explicit log-level attached to them.
3. Sets the lowest possible log-level configuration for the console log-level.

4. Sets default value for the console log-level at boot time.

   Each of these values above defines a different rule for handling error messages.

Procedure

1. Under the KDUMP field, enable kdump if not already enabled.

   
2. Under Kdump Memory Reservation, select Manual` if you must customize the memory reserve.

3. Under KDUMP field, in Memory To Be Reserved (MB), set the required memory reserve for kdump.

   

Procedure

1. Check whether kdump is installed on your system:

   # rpm -q kexec-tools

   Output if the package is installed:

   kexec-tools-2.0.17-11.el8.x86_64

   Output if the package is not installed:

   package kexec-tools is not installed

2. Install kdump and other necessary packages by:

   # dnf install kexec-tools

Procedure

1. Prepare the crashkernel= option.

   • For example, to reserve 128 MB of memory, use the following:

     crashkernel=128M

   • Alternatively, you can set the amount of reserved memory to a variable depending on the total amount of installed memory. The syntax for memory reservation into a variable is crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

     crashkernel=512M-2G:64M,2G-:128M

     The command reserves 64 MB of memory if the total amount of system memory is in the range of 512 MB and 2 GB. If the total amount of memory is more than 2 GB, the memory reserve is 128 MB.

   • Offset the reserved memory.

     Some systems require to reserve memory with a certain fixed offset because the crashkernel reservation happens early, and you may need to reserve more memory for special usage. When you define an offset, the reserved memory begins there. To offset the reserved memory, use the following syntax:

     crashkernel=128M@16M

     In this example, kdump reserves 128 MB of memory starting at 16 MB (physical address 0x01000000). If you set the offset parameter to 0 or omit entirely, kdump offsets the reserved memory automatically. You can also use this syntax when setting a variable memory reservation. In that case, the offset is always specified last. For example:

     crashkernel=512M-2G:64M,2G-:128M@16M

2. Apply the crashkernel= option to your boot loader configuration:

   # grubby --update-kernel=ALL --args="crashkernel=<value>"

   Replace <value> with the value of the crashkernel= option that you prepared in the previous step.

Procedure

1. As root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.
2. To enable crash dump file compression, execute:

Procedure

1. As root, remove the hash sign (#) from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a desired action.

   failure_action poweroff

Procedure

1. Enable the kdump service:

   # kdumpctl restart

2. Check the status of the kdump service. With the kdumpctl command, you can print the output at the console.

   # kdumpctl status
     kdump:Kdump is operational

   Alternatively, if you use the systemctl command, the output prints in the systemd journal.

3. Initiate a kernel crash to test the kdump configuration. The sysrq-trigger key combination causes the kernel to crash and might reboot the system if required.

   # echo c > /proc/sysrq-trigger

   On a kernel reboot, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file. The default is /var/crash/.

Procedure

1. Display the list of modules that are loaded to the currently running kernel. Select the kernel module that you intend to block from loading.

   $ lsmod
   
   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1

2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file. For example:

   KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

   Also, consider the following example using the modprobe.blacklist=<modules> configuration option:

   KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

3. Restart the kdump service:

   # systemctl restart kdump

Procedure

1. Print the estimate crashkernel= value:

   # *kdumpctl estimate*
   
   Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
      Reserved crashkernel:    256M
      Recommended crashkernel: 652M
   
      Kernel image size:   47M
      Kernel modules size: 8M
      Initramfs size:      20M
      Runtime reservation: 64M
      LUKS required size:  512M
      Large modules: <none>
      WARNING: Current crashkernel size is lower than recommended size 652M.

2. Configure the amount of required memory by increasing the crashkernel= value.
3. Reboot the system.

Procedure

1. In the web console, open the Kernel dump tab and start the kdump service by setting the Kernel crash dump switch to on.

2. Configure the kdump memory usage in the terminal, for example:

   $ sudo grubby --update-kernel ALL --args crashkernel=512M

   Restart the system to apply the changes.

3. In the Kernel dump tab, click Edit at the end of the Crash dump location field.

   

4. Specify the target directory for saving the vmcore dump file:

   • For a local filesystem, select Local Filesystem from the drop-down menu.

     

   • For a remote system by using the SSH protocol, select Remote over SSH from the drop-down menu and specify the following fields:

     • In the Server field, enter the remote server address.
     • In the SSH key field, enter the SSH key location.
     • In the Directory field, enter the target directory.

   • For a remote system by using the NFS protocol, select Remote over NFS from the drop-down menu and specify the following fields:

     • In the Server field, enter the remote server address.
     • In the Export field, enter the location of the shared folder of an NFS server.

     • In the Directory field, enter the target directory.

       Note

       You can reduce the size of the vmcore file by selecting the Compression checkbox.

5. Optional: Display the automation script by clicking View automation script.

   A window with the generated script opens. You can navigate between a shell script and an Ansible playbook generation options tab.

6. Optional: Copy the script by clicking Copy to clipboard.

   You can use this script to apply the same configuration on multiple machines.

Verification

1. Click Test configuration.

   

2. Click Crash system under Test kdump settings.

   Warning

   When you initiate the system crash, the kernel operation stops and results in a system crash with data loss.

Procedure

1. Add the crashkernel= command-line parameter to all installed kernels.

   # grubby --update-kernel=ALL --args="crashkernel=xxM"

   xxM is the required memory in megabytes.

2. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. List the kernels installed on the machine.

   # ls -a /boot/vmlinuz-*
   /boot/vmlinuz-0-rescue-2930657cd0dc43c2b75db480e5e5b4a9
   /boot/vmlinuz-4.18.0-330.el8.x86_64
   /boot/vmlinuz-4.18.0-330.rt7.111.el8.x86_64

2. Add a specific kdump kernel to the system’s Grand Unified Bootloader (GRUB) configuration.

   For example:

   # grubby --update-kernel=vmlinuz-4.18.0-330.el8.x86_64 --args="crashkernel=xxM"

   xxM is the required memory reserve in megabytes.

3. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. To stop the kdump service in the current session:

   # systemctl stop kdump.service

2. To disable the kdump service:

   # systemctl disable kdump.service

Procedure

1. To configure final_action, edit the /etc/kdump.conf file and add one of the following options:

   • final_action reboot
   • final_action halt
   • final_action poweroff

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure:

1. To configure an action to take if the dump fails, edit the /etc/kdump.conf file and specify one of the failure_action options:

   • failure_action reboot
   • failure_action halt
   • failure_action poweroff
   • failure_action shell
   • failure_action dump_to_rootfs

2. Restart the kdump service for the changes to take effect.

   # kdumpctl restart

Procedure

1. Install and configure kdump.

2. Enable the fadump=on kernel option:

   # grubby --update-kernel=ALL --args="fadump=on"

3. (Optional) If you want to specify reserved boot memory instead of using the defaults, enable the crashkernel=xxM option, where xx is the amount of the memory required in megabytes:

   # grubby --update-kernel=ALL --args="crashkernel=xxM fadump=on"

   Important

   When specifying boot configuration options, test all boot configuration options before you execute them. If the kdump kernel fails to boot, increase the value specified in crashkernel= argument gradually to set an appropriate value.

Procedure

1. Add or edit the following lines in the /etc/sysctl.conf file to ensure that kdump starts as expected for sadump:

   kernel.panic=0
   kernel.unknown_nmi_panic=1

   Warning

   In particular, ensure that after kdump, the system does not reboot. If the system reboots after kdump has failed to save the vmcore file, then it is not possible to invoke the sadump.

2. Set the failure_action parameter in /etc/kdump.conf appropriately as halt or shell.

   failure_action shell

Procedure

1. Enable the relevant repositories:

   # subscription-manager repos --enable baseos repository

   # subscription-manager repos --enable appstream repository

   # subscription-manager repos --enable rhel-8-for-x86_64-baseos-debug-rpms

2. Install the crash package:

   # yum install crash

3. Install the kernel-debuginfo package:

   # yum install kernel-debuginfo

   The package kernel-debuginfo will correspond to the running kernel and provides the data necessary for the dump analysis.

Procedure

1. To start the crash utility, pass the following two necessary parameters:

   • The debug-info (a decompressed vmlinuz image), for example /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux provided through a specific kernel-debuginfo package.

   • The actual vmcore file, for example /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     The resulting crash command will be as follows:

     # crash /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     Use the same <kernel> version that was captured by kdump.

     Example 20.1. Running the crash utility

     The following example shows analyzing a core dump created on October 6 2018 at 14:05 PM, using the 4.18.0-5.el8.x86_64 kernel.

     ...
     WARNING: kernel relocated [202MB]: patching 90160 gdb minimal_symbol values
     
           KERNEL: /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux
         DUMPFILE: /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore  [PARTIAL DUMP]
             CPUS: 2
             DATE: Sat Oct  6 14:05:16 2018
           UPTIME: 01:03:57
     LOAD AVERAGE: 0.00, 0.00, 0.00
            TASKS: 586
         NODENAME: localhost.localdomain
          RELEASE: 4.18.0-5.el8.x86_64
          VERSION: #1 SMP Wed Aug 29 11:51:55 UTC 2018
          MACHINE: x86_64  (2904 Mhz)
           MEMORY: 2.9 GB
            PANIC: "sysrq: SysRq : Trigger a crash"
              PID: 10635
          COMMAND: "bash"
             TASK: ffff8d6c84271800  [THREAD_INFO: ffff8d6c84271800]
              CPU: 1
            STATE: TASK_RUNNING (SYSRQ)
     
     crash>

2. To exit the interactive prompt and stop crash, type exit or q.

   Example 20.2. Exiting the crash utility

   crash> exit
   ~]#

Procedure

1. Access the Kernel Oops Analyzer tool.

2. To diagnose a kernel crash issue, upload a kernel oops log generated in vmcore.

   • Alternatively you can also diagnose a kernel crash issue by providing a text message or a vmcore-dmesg.txt as an input.

     
3. Click DETECT to compare the oops message based on information from the makedumpfile against known solutions.

Procedure

1. Verify that the kdump service is enabled and active:

   # systemctl is-enabled kdump.service && systemctl is-active kdump.service
   enabled
   active

   If kdump is not enabled and running, set all required configurations and verify that kdump service is enabled.

2. Rebuild the initramfs image of the booting kernel with the early kdump functionality:

   # dracut -f --add earlykdump

3. Add the rd.earlykdump kernel command line parameter:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="rd.earlykdump"

4. Reboot the system to reflect the changes

   # reboot

1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Procedure

1. Optionally, check your kernel version:

   # uname -r
   4.18.0-94.el8.x86_64

2. Search for a live patching package that corresponds to the version of your kernel:

   # yum search $(uname -r)

3. Install the live patching package:

   # yum install "kpatch-patch = $(uname -r)"

   The command above installs and applies the latest cumulative live patches for that specific kernel only.

   If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-4_18_0-94-0-0.el8.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # yum list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64         4.18.0-240.10.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
   kernel-core.x86_64         4.18.0-240.15.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
   ...
   
   # uname -r
   4.18.0-240.10.1.el8_3.x86_64

2. Install the kpatch-dnf plugin:

   # yum install kpatch-dnf

3. Enable automatic subscription to kernel live patches:

   # yum kpatch auto
   Updating Subscription Management repositories.
   Last metadata expiration check: 19:10:26 ago on Wed 10 Mar 2021 04:08:06 PM CET.
   Dependencies resolved.
   ==================================================
    Package                             Architecture
   ==================================================
   Installing:
    kpatch-patch-4_18_0-240_10_1        x86_64
    kpatch-patch-4_18_0-240_15_1        x86_64
   
   Transaction Summary
   ===================================================
   Install  2 Packages
   …​

   This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

   In the future, when you update the kernel, live patches will automatically be installed during the new kernel installation process.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-4_18_0-240-0-0.el8.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # yum list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64         4.18.0-240.10.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
   kernel-core.x86_64         4.18.0-240.15.1.el8_3           @rhel-8-for-x86_64-baseos-rpms
   ...
   
   # uname -r
   4.18.0-240.10.1.el8_3.x86_64

2. Disable automatic subscription to kernel live patches:

   # yum kpatch manual
   Updating Subscription Management repositories.

Procedure

1. Select the live patching package.

   # yum list installed | grep kpatch-patch
   kpatch-patch-4_18_0-94.x86_64        1-1.el8        @@commandline
   …​

   The example output above lists live patching packages that you installed.

2. Remove the live patching package.

   # yum remove kpatch-patch-4_18_0-94.x86_64

   When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

3. Reboot your system.

4. Verify that the live patching package has been removed.

   # yum list installed | grep kpatch-patch

   The command displays no output if the package has been successfully removed.

5. Optionally, verify that the kernel live patching solution is disabled.

   # kpatch list
   Loaded patch modules:

   The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Procedure

1. Select a kernel patch module:

   # kpatch list
   Loaded patch modules:
   kpatch_4_18_0_94_1_1 [enabled]
   
   Installed patch modules:
   kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)
   …​

2. Uninstall the selected kernel patch module.

   # kpatch uninstall kpatch_4_18_0_94_1_1
   uninstalling kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)

   • Note that the uninstalled kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_4_18_0_94_1_1 [enabled]
     
     Installed patch modules:
     <NO_RESULT>

     When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

3. Reboot your system.

4. Optionally, verify that the kernel patch module has been uninstalled.

   # kpatch list
   Loaded patch modules:
   …​

   The example output above shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Procedure

1. Verify kpatch.service is enabled.

   # systemctl is-enabled kpatch.service
   enabled

2. Disable kpatch.service.

   # systemctl disable kpatch.service
   Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

   • Note that the applied kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_4_18_0_94_1_1 [enabled]
     
     Installed patch modules:
     kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)

3. Reboot your system.

4. Optionally, verify the status of kpatch.service.

   # systemctl status kpatch.service
   ● kpatch.service - "Apply kpatch kernel patches"
      Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
      Active: inactive (dead)

   The example output testifies that kpatch.service has been disabled and is not running. Thereby, the kernel live patching solution is not active.

5. Verify that the kernel patch module has been unloaded.

   # kpatch list
   Loaded patch modules:
   <NO_RESULT>
   
   Installed patch modules:
   kpatch_4_18_0_94_1_1 (4.18.0-94.el8.x86_64)

   The example output above shows that a kernel patch module is still installed but the kernel is not patched.

Procedure

1. Identify the process ID (PID) of the application that you want to restrict in CPU consumption:

   # top
   top - 11:34:09 up 11 min,  1 user,  load average: 0.51, 0.27, 0.22
   Tasks: 267 total,   3 running, 264 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 49.0 us,  3.3 sy,  0.0 ni, 47.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
   MiB Mem :   1826.8 total,    303.4 free,   1046.8 used,    476.5 buff/cache
   MiB Swap:   1536.0 total,   1396.0 free,    140.0 used.    616.4 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  99.3   0.1   0:32.71 sha1sum
    5760 jdoe      20   0 3603868 205188  64196 S   3.7  11.0   0:17.19 gnome-shell
    6448 jdoe      20   0  743648  30640  19488 S   0.7   1.6   0:02.73 gnome-terminal-
       1 root      20   0  245300   6568   4116 S   0.3   0.4   0:01.87 systemd
     505 root      20   0       0      0      0 I   0.3   0.0   0:00.75 kworker/u4:4-events_unbound
   ...

   This example output of the top program reveals that illustrative application sha1sum with PID 6955 consumes a lot of CPU resources.

2. Create a sub-directory in the cpu resource controller directory:

   # mkdir /sys/fs/cgroup/cpu/Example/

   This directory represents a control group, where you can place specific processes and apply certain CPU limits to the processes. At the same time, a number of cgroups-v1 interface files and cpu controller-specific files will be created in the directory.

3. Optional: Inspect the newly created control group:

   # ll /sys/fs/cgroup/cpu/Example/
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.clone_children
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.procs
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_all
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_user
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_user
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_quota_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_runtime_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.shares
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpu.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 notify_on_release
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 tasks

   This example output shows files, such as cpuacct.usage, cpu.cfs._period_us, that represent specific configurations and/or limits, which can be set for processes in the Example control group. Note that the respective file names are prefixed with the name of the control group controller to which they belong.

   By default, the newly created control group inherits access to the system’s entire CPU resources without a limit.

4. Configure CPU limits for the control group:

   # echo "1000000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us
   # echo "200000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us

   • The cpu.cfs_period_us file represents a period of time in microseconds (µs, represented here as "us") for how frequently a control group’s access to CPU resources should be reallocated. The upper limit is 1 000 000 microsecond and the lower limit is 1 000 microseconds.

   • The cpu.cfs_quota_us file represents the total amount of time in microseconds for which all processes collectively in a control group can run during one period (as defined by cpu.cfs_period_us). When processes in a control group, during a single period, use up all the time specified by the quota, they are throttled for the remainder of the period and not allowed to run until the next period. The lower limit is 1 000 microseconds.

     The example commands above set the CPU time limits so that all processes collectively in the Example control group will be able to run only for 0.2 seconds (defined by cpu.cfs_quota_us) out of every 1 second (defined by cpu.cfs_period_us).

5. Optional: Verify the limits:

   # cat /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us
   1000000
   200000

6. Add the application’s PID to the Example control group:

   # echo "6955" > /sys/fs/cgroup/cpu/Example/cgroup.procs

   This command ensures that a specific application becomes a member of the Example control group and hence does not exceed the CPU limits configured for the Example control group. The PID should represent an existing process in the system. The PID 6955 here was assigned to process sha1sum /dev/zero &, used to illustrate the use case of the cpu controller.

Verification

1. Verify that the application runs in the specified control group:

   # cat /proc/6955/cgroup
   12:cpuset:/
   11:hugetlb:/
   10:net_cls,net_prio:/
   9:memory:/user.slice/user-1000.slice/user@1000.service
   8:devices:/user.slice
   7:blkio:/
   6:freezer:/
   5:rdma:/
   4:pids:/user.slice/user-1000.slice/user@1000.service
   3:perf_event:/
   2:cpu,cpuacct:/Example
   1:name=systemd:/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service

   This example output shows that the process of the desired application runs in the Example control group, which applies CPU limits to the application’s process.

2. Identify the current CPU consumption of your throttled application:

   # top
   top - 12:28:42 up  1:06,  1 user,  load average: 1.02, 1.02, 1.00
   Tasks: 266 total,   6 running, 260 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 11.0 us,  1.2 sy,  0.0 ni, 87.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.2 st
   MiB Mem :   1826.8 total,    287.1 free,   1054.4 used,    485.3 buff/cache
   MiB Swap:   1536.0 total,   1396.7 free,    139.2 used.    608.3 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  20.6   0.1  47:11.43 sha1sum
    5760 jdoe      20   0 3604956 208832  65316 R   2.3  11.2   0:43.50 gnome-shell
    6448 jdoe      20   0  743836  31736  19488 S   0.7   1.7   0:08.25 gnome-terminal-
     505 root      20   0       0      0      0 I   0.3   0.0   0:03.39 kworker/u4:4-events_unbound
    4217 root      20   0   74192   1612   1320 S   0.3   0.1   0:01.19 spice-vdagentd
   ...

   Note that the CPU consumption of the PID 6955 has decreased from 99% to 20%.

Procedure

1. Identify the process IDs (PIDs) of applications whose CPU consumption you want to constrict:

   # top
   Tasks: 104 total,   3 running, 101 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 17.6 us, 81.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.8 hi,  0.0 si,  0.0 st
   MiB Mem :   3737.4 total,   3312.7 free,    133.3 used,    291.4 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3376.1 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     34578 root      20   0   18720   1756   1468 R  99.0   0.0   0:31.09 sha1sum
     34579 root      20   0   18720   1772   1480 R  99.0   0.0   0:30.54 sha1sum
         1 root      20   0  186192  13940   9500 S   0.0   0.4   0:01.60 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd
         3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
         4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
   ...

   The example output reveals that PID 34578 and 34579 (two illustrative applications of sha1sum) consume a lot of resources, namely CPU. Both are example applications used to demonstrate managing the cgroups-v2 functionality.

2. Verify that the cpu and cpuset controllers are available in the /sys/fs/cgroup/cgroup.controllers file:

   # cat /sys/fs/cgroup/cgroup.controllers
   cpuset cpu io memory hugetlb pids rdma

3. Enable CPU-related controllers:

   # echo "+cpu" >> /sys/fs/cgroup/cgroup.subtree_control
   # echo "+cpuset" >> /sys/fs/cgroup/cgroup.subtree_control

   These commands enable the cpu and cpuset controllers for the immediate children groups of the /sys/fs/cgroup/ root control group. A child group is where you can specify processes and apply control checks to each of the processes based on your criteria.

   Users can read the contents of the cgroup.subtree_control file at any level to get an idea of what controllers are going to be available for enablement in the immediate child group.

   Note

   By default, the /sys/fs/cgroup/cgroup.subtree_control file in the root control group contains memory and pids controllers.

4. Create the /sys/fs/cgroup/Example/ directory:

   # mkdir /sys/fs/cgroup/Example/

   The /sys/fs/cgroup/Example/ directory defines a child group. Also, the previous step enabled the cpu and cpuset controllers for this child group.

   When you create the /sys/fs/cgroup/Example/ directory, some cgroups-v2 interface files and cpu and cpuset controller-specific files are automatically created in the directory. The /sys/fs/cgroup/Example/ directory contains also controller-specific files for the memory and pids controllers.

5. Optionally, inspect the newly created child control group:

   # ll /sys/fs/cgroup/Example/
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.max.depth
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.max.descendants
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.procs
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.stat
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.subtree_control
   …​
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cpuset.cpus
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cpuset.cpus.effective
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cpuset.cpus.partition
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cpuset.mems
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cpuset.mems.effective
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cpu.stat
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cpu.weight
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cpu.weight.nice
   …​
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 memory.events.local
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.high
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.low
   …​
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.current
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 pids.max

   The example output shows files such as cpuset.cpus and cpu.max. These files are specific to the cpuset and cpu controllers. The cpuset and cpu controllers are manually enabled for the root’s (/sys/fs/cgroup/) direct child control groups using the /sys/fs/cgroup/cgroup.subtree_control file.

   The directory also includes general cgroup control interface files such as cgroup.procs or cgroup.controllers, which are common to all control groups, regardless of enabled controllers.

   The files such as memory.high and pids.max relate to the memory and pids controllers, which are in the root control group (/sys/fs/cgroup/), and are always enabled by default.

   By default, the newly created child group inherits access to all of the system’s CPU and memory resources, without any limits.

6. Enable the CPU-related controllers in /sys/fs/cgroup/Example/ to obtain controllers that are relevant only to CPU:

   # echo "+cpu" >> /sys/fs/cgroup/Example/cgroup.subtree_control
   # echo "+cpuset" >> /sys/fs/cgroup/Example/cgroup.subtree_control

   These commands ensure that the immediate child control group will only have controllers relevant to regulate the CPU time distribution - not to memory or pids controllers.

7. Create the /sys/fs/cgroup/Example/tasks/ directory:

   # mkdir /sys/fs/cgroup/Example/tasks/

   The /sys/fs/cgroup/Example/tasks/ directory defines a child group with files that relate purely to cpu and cpuset controllers.

8. Optionally, inspect another child control group:

   # ll /sys/fs/cgroup/Example/tasks
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.depth
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.descendants
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.procs
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.subtree_control
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.threads
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.type
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.max
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.effective
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.partition
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems.effective
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpu.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight.nice
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 io.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 memory.pressure

9. Ensure the processes that you want to control for CPU time compete on the same CPU:

   # echo "1" > /sys/fs/cgroup/Example/tasks/cpuset.cpus

   The previous command ensures that the processes you will place in the Example/tasks child control group, compete on the same CPU. This setting is important for the cpu controller to activate.

   Important

   The cpu controller is only activated if the relevant child control group has at least 2 processes which compete for time on a single CPU.

Verification

1. Optional: ensure that the CPU-related controllers are enabled for the immediate children cgroups:

   # cat /sys/fs/cgroup/cgroup.subtree_control /sys/fs/cgroup/Example/cgroup.subtree_control
   cpuset cpu memory pids
   cpuset cpu

2. Optional: ensure the processes that you want to control for CPU time compete on the same CPU:

   # cat /sys/fs/cgroup/Example/tasks/cpuset.cpus
   1

Procedure

1. Configure CPU bandwidth to achieve resource restrictions within the control group:

   # echo "200000 1000000" > /sys/fs/cgroup/Example/tasks/cpu.max

   The first value is the allowed time quota in microseconds for which all processes collectively in a child group can run during one period. The second value specifies the length of the period.

   During a single period, when processes in a control group collectively exhaust the time specified by this quota, they are throttled for the remainder of the period and not allowed to run until the next period.

   This command sets CPU time distribution controls so that all processes collectively in the /sys/fs/cgroup/Example/tasks child group can run on the CPU for only 0.2 seconds of every 1 second. That is, one fifth of each second.

2. Optionally, verify the time quotas:

   # cat /sys/fs/cgroup/Example/tasks/cpu.max
   200000 1000000

3. Add the applications' PIDs to the Example/tasks child group:

   # echo "34578" > /sys/fs/cgroup/Example/tasks/cgroup.procs
   # echo "34579" > /sys/fs/cgroup/Example/tasks/cgroup.procs

   The example commands ensure that desired applications become members of the Example/tasks child group and do not exceed the CPU time distribution configured for this child group.

Verification

1. Verify that the applications run in the specified control group:

   # cat /proc/34578/cgroup /proc/34579/cgroup
   0::/Example/tasks
   0::/Example/tasks

   The output above shows the processes of the specified applications that run in the Example/tasks child group.

2. Inspect the current CPU consumption of the throttled applications:

   # top
   top - 11:13:53 up 23:10,  1 user,  load average: 0.26, 1.33, 1.66
   Tasks: 104 total,   3 running, 101 sleeping,   0 stopped,   0 zombie
   %Cpu(s):  3.0 us,  7.0 sy,  0.0 ni, 89.5 id,  0.0 wa,  0.2 hi,  0.2 si,  0.2 st
   MiB Mem :   3737.4 total,   3312.6 free,    133.4 used,    291.4 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3376.0 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     34578 root      20   0   18720   1756   1468 R  10.0   0.0  37:36.13 sha1sum
     34579 root      20   0   18720   1772   1480 R  10.0   0.0  37:41.22 sha1sum
         1 root      20   0  186192  13940   9500 S   0.0   0.4   0:01.60 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd
         3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
         4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
   ...

   Notice that the CPU consumption for the PID 34578 and PID 34579 has decreased to 10%. The Example/tasks child group regulates its processes to 20% of the CPU time collectively. Since there are 2 processes in the control group, each can utilize 10% of the CPU time.

Procedure

1. Configure desired CPU weights to achieve resource restrictions within the control groups:

   # echo "150" > /sys/fs/cgroup/Example/g1/cpu.weight
   # echo "100" > /sys/fs/cgroup/Example/g2/cpu.weight
   # echo "50" > /sys/fs/cgroup/Example/g3/cpu.weight

2. Add the applications' PIDs to the g1, g2, and g3 child groups:

   # echo "33373" > /sys/fs/cgroup/Example/g1/cgroup.procs
   # echo "33374" > /sys/fs/cgroup/Example/g2/cgroup.procs
   # echo "33377" > /sys/fs/cgroup/Example/g3/cgroup.procs

   The example commands ensure that desired applications become members of the Example/g*/ child cgroups and will get their CPU time distributed as per the configuration of those cgroups.

   The weights of the children cgroups (g1, g2, g3) that have running processes are summed up at the level of the parent cgroup (Example). The CPU resource is then distributed proportionally based on the respective weights.

   As a result, when all processes run at the same time, the kernel allocates to each of them the proportionate CPU time based on their respective cgroup’s cpu.weight file:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~50% (150/300)
   g2	100	~33% (100/300)
   g3	50	~16% (50/300)

   The value of the cpu.weight controller file is not a percentage.

   If one process stopped running, leaving cgroup g2 with no running processes, the calculation would omit the cgroup g2 and only account weights of cgroups g1 and g3:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~75% (150/200)
   g3	50	~25% (50/200)

   Important

   If a child cgroup had multiple running processes, the CPU time allocated to the respective cgroup would be distributed equally to the member processes of that cgroup.

Verification

1. Verify that the applications run in the specified control groups:

   # cat /proc/33373/cgroup /proc/33374/cgroup /proc/33377/cgroup
   0::/Example/g1
   0::/Example/g2
   0::/Example/g3

   The command output shows the processes of the specified applications that run in the Example/g*/ child cgroups.

2. Inspect the current CPU consumption of the throttled applications:

   # top
   top - 05:17:18 up 1 day, 18:25,  1 user,  load average: 3.03, 3.03, 3.00
   Tasks:  95 total,   4 running,  91 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 18.1 us, 81.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
   MiB Mem :   3737.0 total,   3233.7 free,    132.8 used,    370.5 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3373.1 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     33373 root      20   0   18720   1748   1460 R  49.5   0.0 415:05.87 sha1sum
     33374 root      20   0   18720   1756   1464 R  32.9   0.0 412:58.33 sha1sum
     33377 root      20   0   18720   1860   1568 R  16.3   0.0 411:03.12 sha1sum
       760 root      20   0  416620  28540  15296 S   0.3   0.7   0:10.23 tuned
         1 root      20   0  186328  14108   9484 S   0.0   0.4   0:02.00 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthread
   ...

   Note

   We forced all the example processes to run on a single CPU for clearer illustration. The CPU weight applies the same principles also when used on multiple CPUs.

   Notice that the CPU resource for the PID 33373, PID 33374, and PID 33377 was allocated based on the weights, 150, 100, 50, you assigned to the respective child cgroups. The weights correspond to around 50%, 33%, and 16% allocation of CPU time for each application.