Installing RHEL 9 for Real Time

• The blue points in this graph represent the system response time (in microseconds) of machines running a tuned RHEL kernel.
• The green points in the graph represent the system response time of machines running a tuned real-time kernel.

Prerequisites

• The latest version of RHEL 9 is installed on an AMD64 or Intel64 system. The real-time kernel runs on AMD64 and Intel 64 (also known as x86_64) server platforms that are certified to run Red Hat Enterprise Linux.
• Your machine is registered and RHEL is attached to a RHEL for Real Time subscription.

• Ensure that the base platform is properly tuned and the system BIOS parameters are adjusted.

  Warning

  Failure to perform the prerequisite tasks before installing the real-time kernel might prevent a consistent performance from a RHEL for Real Time kernel deployment.

Verification

• Check the installation location and verify that the components have been installed successfully.

  # rpm -ql realtime-setup
  /etc/security/limits.d/realtime.conf
  /etc/sysconfig/realtime-setup
  /etc/udev/rules.d/99-rhel-rt.rules
  /usr/bin/realtime-setup
  /usr/bin/rt-setup-kdump
  /usr/bin/slub_cpu_partial_off
  /usr/lib/.build-id
  /usr/lib/.build-id/a4
  /usr/lib/.build-id/a4/da77908aa4c6f048939f3267f1c552c456d117
  /usr/lib/systemd/system/rt-entsk.service
  /usr/lib/systemd/system/rt-setup.service
  /usr/sbin/kernel-is-rt
  /usr/sbin/rt-entsk

  Copy to Clipboard

• kernel-rt package, which is the RHEL for Real Time kernel package.
• RHEL for Real Time kernel test packages, which contains test programs for the real-time kernel.
• RHEL for Real Time debugging packages, which are for debugging and code tracing.

• To achieve optimal low-latency determinism, you perform RHEL for Real Time specific system tuning.
• You know about the module compatibility of the real-time kernel and the standard kernel.
• To enable kdump, you must configure RHEL for Real Time to provide crash dump information by enabling kexec/kdump.
• Verify that the Real Time kernel is the default kernel.

• EMC Powerpath
• NVidia graphics
• Advanced storage adapter configuration utilities from Qlogic

Procedure

• To view the default kernel:

  # grubby --default-kernel
  
  /boot/vmlinuz-kernel-rt-5.14.0-70.13.1.rt21.83.el9_0

  Copy to Clipboard

  The rt in the output of the command shows that the default kernel is a real time kernel.

Procedure

• To show which kernel the system is currently running.

  ~]# uname -a
  Linux rt-server.example.com 4.18.0-80.rt9.138.el8.x86_64 …

  Copy to Clipboard
  Note

  When the system receives a minor update, for example, from 8.3 to 8.4, the default kernel might automatically change from the Real Time kernel back to the standard kernel.

Verification

• To verify kernel-rt is the default kernel, enter the following command:

  # grubby --default-kernel
  /boot/vmlinuz-5.14.0-284.11.1.rt14.296.el9_2.x86_64.x86_64+rt

  Copy to Clipboard

Prerequisites

• An active RHEL subscription.
• A repository containing the kexec-tools package for your system CPU architecture.
• Fulfilled requirements for kdump configurations and targets. For details, see Supported kdump configurations and targets.

Procedure

• Enter the following command to generate a memory usage report:

  # makedumpfile --mem-usage /proc/kcore
  
  
  TYPE        PAGES    EXCLUDABLE    DESCRIPTION
  -------------------------------------------------------------
  ZERO          501635      yes        Pages filled with zero
  CACHE         51657       yes        Cache pages
  CACHE_PRIVATE 5442        yes        Cache pages + private
  USER          16301       yes        User process pages
  FREE          77738211    yes        Free pages
  KERN_DATA     1333192     no         Dumpable kernel data

  Copy to Clipboard

Prerequisites

• You have root permissions on the system.
• You have fulfilled kdump requirements for configurations and targets. For details, see Supported kdump configurations and targets.
• You have installed the zipl utility if it is the IBM Z system.

Prerequisites

• You have root permissions on the system.
• Fulfilled requirements for kdump configurations and targets. For details, see Supported kdump configurations and targets.

Procedure

• To store the crash dump file in /var/crash/ directory of the local file system, edit the /etc/kdump.conf file and specify the path:

  path /var/crash

  Copy to Clipboard

  The option path /var/crash represents the path to the file system in which kdump saves the crash dump file.

  Note
  • When you specify a dump target in the /etc/kdump.conf file, then the path is relative to the specified dump target.
  • When you do not specify a dump target in the /etc/kdump.conf file, then the path represents the absolute path from the root directory.

  Depending on the file system mounted in the current system, the dump target and the adjusted dump path are configured automatically.

• To secure the crash dump file and the accompanying files produced by kdump, you should set up proper attributes for the target destination directory, such as user permissions and SELinux contexts. Additionally, you can define a script, for example kdump_post.sh in the kdump.conf file as follows:

  kdump_post <path_to_kdump_post.sh>

  Copy to Clipboard

  The kdump_post directive specifies a shell script or a command that executes after kdump has completed capturing and saving a crash dump to the specified destination. You can use this mechanism to extend the functionality of kdump to perform actions including the adjustments in file permissions.

• The kdump target configuration

• To change the local directory for saving the crash dump, edit the /etc/kdump.conf configuration file as a root user:

  1. Remove the hash sign (#) from the beginning of the #path /var/crash line.

  2. Replace the value with the intended directory path. For example:

     path /usr/local/cores

     Copy to Clipboard
     Important

     In RHEL 9, the directory defined as the kdump target using the path directive must exist when the kdump systemd service starts to avoid failures. Unlike in earlier versions of RHEL, the directory is no longer created automatically if it does not exist when the service starts.

• To write the file to a different partition, edit the /etc/kdump.conf configuration file:

  1. Remove the hash sign (#) from the beginning of the #ext4 line, depending on your choice.

     • device name (the #ext4 /dev/vg/lv_kdump line)
     • file system label (the #ext4 LABEL=/boot line)
     • UUID (the #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 line)

  2. Change the file system type and the device name, label or UUID, to the required values. The correct syntax for specifying UUID values is both UUID="correct-uuid" and UUID=correct-uuid. For example:

     ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937

     Copy to Clipboard
     Important

     It is recommended to specify storage devices by using a LABEL= or UUID=. Disk device names such as /dev/sda3 are not guaranteed to be consistent across reboot.

     When you use Direct Access Storage Device (DASD) on IBM Z hardware, ensure the dump devices are correctly specified in /etc/dasd.conf before proceeding with kdump.

• To write the crash dump directly to a device, edit the /etc/kdump.conf configuration file:

  1. Remove the hash sign (#) from the beginning of the #raw /dev/vg/lv_kdump line.

  2. Replace the value with the intended device name. For example:

     raw /dev/sdb1

     Copy to Clipboard

• To store the crash dump to a remote machine by using the NFS protocol:

  1. Remove the hash sign (#) from the beginning of the #nfs my.server.com:/export/tmp line.

  2. Replace the value with a valid hostname and directory path. For example:

     nfs penguin.example.com:/export/cores

     Copy to Clipboard

  3. Restart the kdump service for the changes to take effect:

     sudo systemctl restart kdump.service

     Copy to Clipboard
     Note

     While using the NFS directive to specify the NFS target, kdump.service automatically attempts to mount the NFS target to check the disk space. There is no need to mount the NFS target in advance. To prevent kdump.service from mounting the target, use the dracut_args --mount directive in kdump.conf. This will enable kdump.service to call the dracut utility with the --mount argument to specify the NFS target.

• To store the crash dump to a remote machine by using the SSH protocol:

  1. Remove the hash sign (#) from the beginning of the #ssh user@my.server.com line.
  2. Replace the value with a valid username and hostname.

  3. Include your SSH key in the configuration.

     1. Remove the hash sign from the beginning of the #sshkey /root/.ssh/kdump_id_rsa line.

     2. Change the value to the location of a key valid on the server you are trying to dump to. For example:

        ssh john@penguin.example.com
        sshkey /root/.ssh/mykey

        Copy to Clipboard

• Compressing the size of a crash dump file and copying only necessary pages by using various dump levels.
• Excluding unnecessary crash dump pages.
• Filtering the page types to be included in the crash dump.

Options

• -c, -l or -p: specify compress dump file format by each page using either, zlib for -c option, lzo for -l option or snappy for -p option.
• -d (dump_level): excludes pages so that they are not copied to the dump file.
• --message-level : specify the message types. You can restrict outputs printed by specifying message_level with this option. For example, specifying 7 as message_level prints common messages and error messages. The maximum value of message_level is 31.

Prerequisites

• You have root permissions on the system.
• Fulfilled requirements for kdump configurations and targets. For details, see Supported kdump configurations and targets.

• To compress a crash dump file by using -c:

  core_collector makedumpfile -c -d 31 --message-level 1

  Copy to Clipboard

• To compress a crash dump file by using -p:

  core_collector makedumpfile -p -d 31 --message-level 1

  Copy to Clipboard

Prerequisites

• Root permissions.
• Fulfilled requirements for kdump configurations and targets. For details, see Supported kdump configurations and targets.

• Package upgrades.
• Hardware level changes, for example, storage or networking changes.
• Firmware upgrades.
• New installation and application upgrades that include third party modules.
• If you use the hot-plugging mechanism to add more memory on hardware that support this mechanism.
• After you make changes in the /etc/kdump.conf or /etc/sysconfig/kdump file.

Prerequisites

• You have root permissions on the system.
• You have saved all important data. The commands to test kdump cause the kernel to crash with loss of data.
• You have scheduled significant machine maintenance time depending on the system architecture.

• vmcore - main kernel memory dump file containing system memory at the time of the crash. It includes data as per the configuration of the core_collector program specified in kdump configuration. By default the kernel data structures, process information, stack traces, and other diagnostic information.
• vmcore-dmesg.txt - contents of the kernel ring buffer log (dmesg) from the primary kernel that panicked.
• kexec-dmesg.log - has kernel and system log messages from the execution of the secondary kexec kernel that collects the vmcore data.

Prerequisites

• You have root permissions on the system.
• You have completed kdump requirements for configurations and targets. See Supported kdump configurations and targets.
• All configurations for installing kdump are set up as required.

• rd.driver.blacklist=<modules>
• modprobe.blacklist=<modules>

Prerequisites

• You have root permissions on the system.

Prerequisites

• You have administrator privileges.

Verification

• Check that the kdump service is running:

  # systemctl status kdump.service
  
  ○ kdump.service - Crash recovery kernel arming
       Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: disabled)
       Active: active (live)

  Copy to Clipboard

Prerequisites

• You have administrator privileges.

Verification

• Check that the kdump service is running.

  # systemctl status kdump.service
  
  ○ kdump.service - Crash recovery kernel arming
       Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: disabled)
       Active: active (live)

  Copy to Clipboard

Prerequisites

• Fulfilled requirements for kdump configurations and targets. For details, see Supported kdump configurations and targets.
• All configurations for installing kdump are set up according to your needs. For details, see Installing kdump.