Chapter 4. Configuring kdump on the command line
The memory for kdump is reserved during the system boot. You can configure the memory size in the system’s Grand Unified Bootloader (GRUB) configuration file. The memory size depends on the crashkernel= value specified in the configuration file and the size of the physical memory of the system.
4.1. Estimating the kdump size Copy linkLink copied to clipboard!
To plan and build your kdump environment effectively, estimate the space required by the crash dump file.
The makedumpfile --mem-usage command estimates the space required by the crash dump file. It generates a memory usage report. The report helps you decide the dump level and the pages that are safe to exclude.
Procedure
Enter the following command to generate a memory usage report:
# makedumpfile --mem-usage /proc/kcore TYPE PAGES EXCLUDABLE DESCRIPTION ------------------------------------------------------------- ZERO 501635 yes Pages filled with zero CACHE 51657 yes Cache pages CACHE_PRIVATE 5442 yes Cache pages + private USER 16301 yes User process pages FREE 77738211 yes Free pages KERN_DATA 1333192 no Dumpable kernel dataImportantThe
makedumpfile --mem-usagecommand reports required memory in pages. This means that you must calculate the size of memory in use against the kernel page size.
4.2. Configuring kdump memory usage Copy linkLink copied to clipboard!
Configure the memory reservation for kdump. The kdump-utils package provides default crashkernel= values that you can use as a reference when setting memory manually.
The automatic memory allocation for kdump also varies based on the system hardware architecture and available memory size. For example, on AMD64 and Intel 64-bit architectures, the default value for the crashkernel= parameter works only when the available memory is more than 2 GB. The kdump-utils utility configures the following default memory reserves on AMD64 and Intel 64-bit architecture:
crashkernel=2G-64G:256M,64G-:512M
You can also run kdumpctl estimate to get an approximate value without triggering a crash. The estimated crashkernel= value might not be an exact one but can serve as a reference to set an appropriate crashkernel= value.
+ Note that the crashkernel=1G-4G:192M,4G-64G:256M,64G:512M option in the boot command line is no longer supported on RHEL 10 and later releases.
The commands to test kdump configuration cause the kernel to crash with data loss. Follow the instructions with care. Do not use an active production system to test the kdump configuration.
Prerequisites
- You have root permissions on the system.
-
You have fulfilled
kdumprequirements for configurations and targets. For details, see Supported kdump configurations and targets. -
You have installed the
ziplutility if it is the IBM Z system.
Procedure
Configure the default value for crash kernel:
# kdumpctl reset-crashkernel --kernel=ALLWhen configuring the
crashkernel=value, test the configuration by rebooting the system withkdumpenabled. If thekdumpkernel fails to boot, increase the memory size gradually to set an acceptable value.To use a custom
crashkernel=value:Configure the required memory reserve.
crashkernel=192MOptionally, you can set the amount of reserved memory to a variable depending on the total amount of installed memory by using the syntax
crashkernel=<range1>:<size1>,<range2>:<size2>. For example:crashkernel=1G-4G:192M,2G-64G:256MThe example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for
kdump.Optional: Offset the reserved memory.
Some systems must reserve memory at a certain fixed offset. The
crashkernelreservation occurs early during the boot process, and the system uses some memory areas for special purposes. Setting a memory offset ensures that the reserved crash memory does not overlap with these areas, which helps prevent boot failures and ensures reliable crash dump capture. To offset the reserved memory, use the following syntax:crashkernel=192M@16MThe example reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If you offset to 0 or do not specify a value,
kdumpoffsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example,crashkernel=1G-4G:192M,2G-64G:256M@16M.Update the boot loader configuration:
# grubby --update-kernel ALL --args "crashkernel=<custom-value>"The
<custom-value>must contain the customcrashkernel=value that you have configured for the crash kernel.
Reboot for changes to take effect:
# reboot
Verification
Cause the kernel to crash by activating the sysrq key. The address-YYYY-MM-DD-HH:MM:SS/vmcore file is saved to the target location as specified in the /etc/kdump.conf file. If you select the default target location, the vmcore file is saved in the partition mounted under /var/crash/.
Activate the
sysrqkey to boot into thekdumpkernel:# echo c > /proc/sysrq-triggerThe command causes kernel to crash and reboots the kernel if required.
Display the
/etc/kdump.conffile and check if thevmcorefile is saved in the target destination.See the
grubby(8)man page on your system for more information.
4.3. Configuring the kdump target Copy linkLink copied to clipboard!
Configure where kdump stores crash dump files. You can store dumps locally or send them over a network by using Network File System (NFS) or Secure Shell (SSH). Only one storage option can be configured at a time. The default behavior stores crash dumps in the /var/crash/ directory of the local file system.
Prerequisites
- You have root permissions on the system.
-
Fulfilled requirements for
kdumpconfigurations and targets. For details, see Supported kdump configurations and targets.
Procedure
To store the crash dump file in
/var/crash/directory of the local file system, edit the/etc/kdump.conffile and specify the path:path /var/crashThe option
path /var/crashrepresents the path to the file system in whichkdumpsaves the crash dump file.When you specify a dump target in the
/etc/kdump.conffile, the path is relative to the specified dump target. When you do not specify a dump target, the path represents the absolute path from the root directory. Depending on the file system mounted in the current system, the dump target and the adjusted dump path are configured automatically.To secure the crash dump file and the accompanying files produced by
kdump, you must configure proper attributes for the target destination directory. These attributes include user permissions and SELinux contexts. Additionally, you can define a script, for examplekdump_post.shin thekdump.conffile as follows:kdump_post <path_to_kdump_post.sh>The
kdump_postdirective specifies a shell script or a command. This script runs afterkdumphas captured and saved a crash dump to the specified destination. You can use this mechanism to extend the functionality ofkdumpto perform actions including the adjustments in file permissions.Displaying and understanding the
kdumptarget configuration:Show the effective configuration by filtering out comments and empty lines:
# grep -v '^#' /etc/kdump.conf | grep -v '^$'Example output:
ext4 /dev/mapper/vg00-varcrashvol path /var/crash core_collector makedumpfile -c --message-level 1 -d 31The dump target is specified (
ext4 /dev/mapper/vg00-varcrashvol), and, therefore, it is mounted at/var/crash. Thepathoption is also set to/var/crash. Therefore, thekdumpsaves thevmcorefile in the/var/crash/var/crashdirectory.
To change the local directory for saving the crash dump, edit the
/etc/kdump.confconfiguration file as arootuser:-
Remove the hash sign (
#) from the beginning of the#path /var/crashline. Replace the value with the intended directory path. For example:
path /usr/local/coresImportantIn Red Hat Enterprise Linux 10, the directory defined as the
kdumptarget using thepathdirective must exist before thekdumpsystemd service starts. Otherwise, the service fails. The directory is no longer created automatically if it does not exist when the service starts.
-
Remove the hash sign (
To write the file to a different partition, edit the
/etc/kdump.confconfiguration file:Remove the hash sign (
#) from the beginning of the#ext4line, depending on your choice.-
device name (the
#ext4 /dev/vg/lv_kdumpline) -
file system label (the
#ext4 LABEL=/bootline) -
UUID (the
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937line)
-
device name (the
Change the file system type and the device name, label or UUID, to the required values. The correct syntax for specifying UUID values is both
UUID="correct-uuid"andUUID=correct-uuid. For example:ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937ImportantYou must specify storage devices by using a
LABEL=orUUID=. Disk device names such as/dev/sda3are not guaranteed to be consistent across reboot.When you use Direct Access Storage Device (DASD) on IBM Z hardware, verify the dump device configuration. Ensure the dump devices are correctly specified in
/etc/dasd.confbefore proceeding withkdump.
To write the crash dump directly to a device, edit the
/etc/kdump.confconfiguration file:-
Remove the hash sign (
#) from the beginning of the#raw /dev/vg/lv_kdumpline. Replace the value with the intended device name. For example:
raw /dev/sdb1
-
Remove the hash sign (
To store the crash dump to a remote machine by using the
NFSprotocol:-
Remove the hash sign (
#) from the beginning of the#nfs my.server.com:/export/tmpline. Replace the value with a valid hostname and directory path. For example:
nfs penguin.example.com:/export/coresRestart the
kdumpservice for the changes to take effect:$ sudo systemctl restart kdump.serviceNoteWhen you use the NFS directive to specify the NFS target,
kdump.serviceautomatically attempts to mount the target. This mount verifies the available disk space. There is no need to mount the NFS target in advance. To preventkdump.servicefrom mounting the target, use thedracut_args --mountdirective inkdump.conf. This enableskdump.serviceto call thedracututility with the--mountargument to specify the NFS target.
-
Remove the hash sign (
To store the crash dump to a remote machine by using the SSH protocol:
-
Remove the hash sign (
#) from the beginning of the#ssh user@my.server.comline. - Replace the value with a valid username and hostname.
Include your SSH key in the configuration.
-
Remove the hash sign from the beginning of the
#sshkey /root/.ssh/kdump_id_rsaline. Change the value to the location of a key valid on the server you are trying to dump to. For example:
ssh john@penguin.example.com sshkey /root/.ssh/mykey
-
Remove the hash sign from the beginning of the
-
Remove the hash sign (
4.4. Configuring the kdump core collector Copy linkLink copied to clipboard!
To configure the kdump core collector, use the makedumpfile utility. It is the default collector in RHEL and helps reduce crash dump size by compressing data and excluding unnecessary pages.
- Compressing the size of a crash dump file and copying only necessary pages by using various dump levels.
- Excluding unnecessary crash dump pages.
- Filtering the page types to be included in the crash dump.
Crash dump file compression is enabled by default.
If you need to customize the crash dump file compression, follow this procedure.
- Syntax
core_collector makedumpfile -l --message-level 1 -d 31
- Options
-
-c,-lor-p: specify compress dump file format by each page using either,zlibfor-coption,lzofor-loption,snappyfor-poption orzstdfor-zoption. -
-d(dump_level): excludes pages so that they are not copied to the dump file. -
--message-level: specify the message types. You can restrict outputs printed by specifyingmessage_levelwith this option. For example, specifying 7 asmessage_levelprints common messages and error messages. The maximum value ofmessage_levelis 31.
-
Prerequisites
- You have root permissions on the system.
-
Fulfilled requirements for
kdumpconfigurations and targets. For details, see Supported kdump configurations and targets.
Procedure
-
As a
root, edit the/etc/kdump.confconfiguration file. Remove the hash sign ("#") from the beginning of the#core_collector makedumpfile -l --message-level 1 -d 31line. Enter the following command to enable crash dump file compression:
core_collector makedumpfile -l --message-level 1 -d 31The
-loption sets the compressed file format to LZO. The-doption sets the dump level to 31. The--message-leveloption sets the message level to 1. You can also use the-c,-p, or-zoptions to specify other compression formats.See
makedumpfile(8)man page on your system for more information.
4.5. Configuring the kdump default failure responses Copy linkLink copied to clipboard!
To configure the default failure response for kdump, specify an action other than the default reboot. This determines the system behavior when saving the core dump fails.
The additional actions include the following options:
dump_to_rootfs-
Saves the core dump to the
rootfile system. reboot- Reboots the system, losing the core dump in the process.
halt- Stops the system, losing the core dump in the process.
poweroff- Power the system off, losing the core dump in the process.
shell-
Runs a shell session from within the
initramfs, you can record the core dump manually. final_action-
Enables additional operations such as
reboot,halt, andpoweroffafter a successfulkdumpor when shell ordump_to_rootfsfailure action completes. The default isreboot. failure_action-
Specifies the action to perform when a dump might fail in a kernel crash. The default is
reboot.
Prerequisites
- Root permissions.
-
Fulfilled requirements for
kdumpconfigurations and targets. For details, see Supported kdump configurations and targets.
Procedure
-
As a
rootuser, remove the hash sign (#) from the beginning of the#failure_actionline in the/etc/kdump.confconfiguration file. Replace the value with a required action.
failure_action poweroff
4.6. Configuration file for kdump Copy linkLink copied to clipboard!
The configuration file for kdump kernel is /etc/sysconfig/kdump. This file controls the kdump kernel command line parameters. For most configurations, use the default options.
However, in some scenarios you might have to modify certain parameters to control the kdump kernel behavior. For example, you can modify the KDUMP_COMMANDLINE_APPEND option to append the kdump kernel command-line for detailed debugging output. You can also modify the KDUMP_COMMANDLINE_REMOVE option to remove arguments from the kdump command line.
KDUMP_COMMANDLINE_REMOVEThis option removes arguments from the current
kdumpcommand line. It removes parameters that can causekdumperrors orkdumpkernel boot failures. These parameters might have been parsed from the previousKDUMP_COMMANDLINEprocess or inherited from the/proc/cmdlinefile.When this variable is not configured, it inherits all values from the
/proc/cmdlinefile. Configuring this option also provides information that is helpful in debugging an issue.To remove certain arguments, add them to
KDUMP_COMMANDLINE_REMOVEas follows:# KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"KDUMP_COMMANDLINE_APPENDThis option appends arguments to the current command line. These arguments might have been parsed by the previous
KDUMP_COMMANDLINE_REMOVEvariable.For the
kdumpkernel, disabling certain modules such asmce,cgroup,numa,hest_disablecan help prevent kernel errors. These modules can consume a significant part of the kernel memory reserved forkdumpor causekdumpkernel boot failures.To disable memory cgroups on the
kdumpkernel command line, run the command as follows:KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory"See
/etc/sysconfig/kdumpfile for more information.
4.7. Testing the kdump configuration Copy linkLink copied to clipboard!
After configuring kdump, manually test a system crash and ensure that the vmcore file is generated in the defined kdump target. The vmcore file is captured from the context of the freshly booted kernel. Therefore, vmcore has critical information for debugging a kernel crash.
Do not test kdump on active production systems. The commands to test kdump cause the kernel to crash with data loss. Depending on your system architecture, schedule significant maintenance time. The kdump testing might require several reboots with a long boot time.
If the vmcore file is not generated during the kdump test, identify and fix the issues. Then, run the test again to verify successful kdump operation.
If you make any manual system modifications, you must test the kdump configuration at the end of any system modification. For example, if you make any of the following changes, ensure that you test the kdump configuration for optimal kdump performance for:
- Package upgrades.
- Hardware level changes, for example, storage or networking changes.
- Firmware upgrades.
- New installation and application upgrades that include third party modules.
- If you use the hot-plugging mechanism to add more memory on hardware that support this mechanism.
-
After you make changes in the
/etc/kdump.confor/etc/sysconfig/kdumpfile.
Prerequisites
- You have root permissions on the system.
-
You have saved all important data. The commands to test
kdumpcause the kernel to crash with loss of data. - You have scheduled significant machine maintenance time depending on the system architecture.
Procedure
Enable the
kdumpservice:# kdumpctl restartCheck the status of the
kdumpservice with thekdumpctl:# kdumpctl statuskdump:Kdump is operationalOptionally, if you use the
systemctlcommand, the output prints in the systemd journal.Start a kernel crash to test the
kdumpconfiguration. Thesysrq-triggerkey combination causes the kernel to crash and might reboot the system if required.# echo c > /proc/sysrq-triggerOn a kernel reboot, the
address-YYYY-MM-DD-HH:MM:SS/vmcorefile is created at the location you have specified in the/etc/kdump.conffile. The default is/var/crash/.
4.8. Files produced by kdump after system crash Copy linkLink copied to clipboard!
After your system crashes, the kdump service captures the kernel memory in a dump file (vmcore). The service also generates diagnostic files to aid in troubleshooting and postmortem analysis.
Files produced by kdump:
-
vmcore- main kernel memory dump file containing system memory at the time of the crash. It includes data according to the configuration of thecore_collectorprogram specified inkdumpconfiguration. By default, it includes the kernel data structures, process information, stack traces, and other diagnostic information. -
vmcore-dmesg.txt- contents of the kernel ring buffer log (dmesg) from the primary kernel that panicked. -
kexec-dmesg.log- has kernel and system log messages from the execution of the secondarykexeckernel that collects thevmcoredata.
4.9. Enabling and disabling the kdump service Copy linkLink copied to clipboard!
You can configure to enable or disable the kdump functionality on a specific kernel or on all installed kernels. You must routinely test the kdump functionality and validate that it operates correctly.
Prerequisites
- You have root permissions on the system.
-
You have completed
kdumprequirements for configurations and targets. See Supported kdump configurations and targets. -
All configurations for installing
kdumpare set up as required.
Procedure
Enable the
kdumpservice formulti-user.target:# systemctl enable kdump.serviceStart the service in the current session:
# systemctl start kdump.serviceStop the
kdumpservice:# systemctl stop kdump.serviceDisable the
kdumpservice:# systemctl disable kdump.serviceWarningSet
kptr_restrict=1as default, if your environment requirements do not prevent it. Whenkptr_restrictis set to (1) as default, thekdumpctlservice loads the crash kernel. This occurs regardless of whether Kernel Address Space Layout Randomization (KASLR) is enabled.If
kptr_restrictis not set to1and KASLR is enabled, the contents of/proc/kcorefile are generated as all zeros. Thekdumpctlservice fails to access the/proc/kcorefile and load the crash kernel, and thekexec-kdump-howto.txtfile displays a warning message that recommends settingkptr_restrict=1. Verify the following in thesysctl.conffile to ensure thatkdumpctlservice loads the crash kernel:-
The kernel
kptr_restrict=1setting in thesysctl.conffile.
-
The kernel
4.10. Preventing kernel drivers from loading for kdump Copy linkLink copied to clipboard!
To prevent specific kernel drivers from loading in the capture kernel, use the KDUMP_COMMANDLINE_APPEND= variable in /etc/sysconfig/kdump. This stops the kdump initramfs from loading modules, helping to avoid out-of-memory (OOM) errors and other crash kernel failures.
You can append the KDUMP_COMMANDLINE_APPEND= variable by using one of the following configuration options:
-
rd.driver.blacklist=<modules> -
modprobe.blacklist=<modules>
Prerequisites
- You have root permissions on the system.
Procedure
Display the list of modules that are loaded to the currently running kernel. Select the kernel module that you intend to block from loading:
$ lsmodModule Size Used by fuse 126976 3 xt_CHECKSUM 16384 1 ipt_MASQUERADE 16384 1 uinput 20480 1 xt_conntrack 16384 1Update the
KDUMP_COMMANDLINE_APPEND=variable in the/etc/sysconfig/kdumpfile. For example:KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"Also, consider the following example by using the
modprobe.blacklist=<modules>configuration option:KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"Restart the
kdumpservice:# systemctl restart kdumpSee the
dracut.cmdlineman page on your system for more information.
4.11. Running kdump on systems with encrypted disk Copy linkLink copied to clipboard!
When you run a Linux Unified Key Setup (LUKS) encrypted partition, systems require a certain amount of available memory. If the system has less than the required amount of available memory, the cryptsetup utility fails to mount the partition. As a result, capturing the vmcore file to an encrypted target location fails in the second kernel (capture kernel).
The kdumpctl estimate command helps you estimate the amount of memory you need for kdump. kdumpctl estimate prints the recommended crashkernel value, which is the most suitable memory size required for kdump.
The recommended crashkernel value is calculated based on the current kernel size, kernel module, initramfs, and the LUKS encrypted target memory requirement.
If you are using the custom crashkernel= option, kdumpctl estimate prints the LUKS required size value. The value is the memory size required for LUKS encrypted target.
Procedure
Print the estimate
crashkernel=value:# *kdumpctl estimate* Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement Reserved crashkernel: 256M Recommended crashkernel: 652M Kernel image size: 47M Kernel modules size: 8M Initramfs size: 20M Runtime reservation: 64M LUKS required size: 512M Large modules: <none> WARNING: Current crashkernel size is lower than recommended size 652M.-
Configure the amount of required memory by increasing the
crashkernel=value. Reboot the system.
NoteIf the
kdumpservice still fails to save the dump file to the encrypted target, increase thecrashkernel=value as required.