Red Hat SummitChapter 1. Installing RHEL for real time
Many industries and organizations need performance computing and might require low and predictable latency, especially in the financial and telecommunications industries. Latency, or response time, is defined as the time between an event and system response and is generally measured in microseconds (μs).
Basic performance tuning can improve latency for most applications in a Linux environment. For industries that require low, accountable, and predictable latency, Red Hat developed a 'drop-in' kernel replacement. RHEL for Real Time is part of RHEL 10 and offers clients the opportunity to measure, configure, and record latency times.
On 64-bit ARM (aarch64) systems, the RHEL for Real Time kernel (kernel-rt) is available as a Technology Preview. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
1.1. RHEL for Real Time for optimizing latency
RHEL for Real Time is designed to be used on well-tuned systems for applications with extremely high determinism requirements. The kernel offers the vast majority of the improvement in determinism.
For example, in many workloads, thorough system tuning improves consistency of results to an optimum percentage. Before using RHEL for Real Time, perform system tuning of standard RHEL first. This tuning helps determine if standard RHEL meets your objectives.
System tuning is just as important when using the Real Time kernel as it is for the standard kernel. Installing the Real-time kernel on an untuned system is not likely to result in any noticeable benefit. Tuning the standard kernel helps achieve good latency numbers, though not consistently. The Real Time kernel provides the required latency reduction, determinism, and consistency.
Before tuning Real Time kernel systems, ensure that the base platform is properly tuned and the system firmware parameters are adjusted. Failure to perform these tasks might prevent getting consistent performance from a RHEL Real-time deployment.
The objective of the Real-time kernel is to provide consistent, low-latency and determinism, offering predictable response times. The real time kernel requires additional processing resources. To provide reduced latency and ensure the highest-priority task receives more processing time, extra infrastructure is required. This additional infrastructure increases resource usage. This increased resource usage might result in some degradation in overall throughput. The exact amount is very workload dependent, ranging from 0% to 30%.
For typical workloads with kernel latency requirements in the millisecond (ms) range, the standard RHEL kernel is sufficient. However, if your workload has stringent requirements for core kernel features, such as interrupt handling and process scheduling, to have latency in the microsecond (μs) range, using the Real Time kernel is recommended.
1.2. Installing RHEL for Real Time using dnf
To obtain the real-time kernel, you can install it by using the dnf utility, or you can download it along with other RHEL components from the Download Red Hat Enterprise Linux portal.
Prerequisites
- The latest version of RHEL 10 is installed on an AMD64 or Intel 64 system. The real-time kernel runs on AMD64 and Intel 64 (also known as x86_64) server platforms that are certified to run Red Hat Enterprise Linux.
- Your machine is registered and RHEL is attached to a RHEL for Real Time subscription.
- Ensure that the base platform is properly tuned and the system firmware parameters are adjusted. Failure to perform these tasks before installing the real-time kernel might prevent consistent performance from a RHEL for Real Time deployment.
Procedure
Enable the RHEL for Real Time repository.
# subscription-manager repos --enable rhel-10-for-x86_64-rt-rpmsNoteIf you are using an Extended Update Support (EUS) subscription, enable the RHEL for Real Time EUS repository instead:
# subscription-manager repos --enable rhel-10-for-x86_64-rt-eus-rpmsFor more information about EUS, see Red Hat Enterprise Linux Life Cycle.
Install the RHEL for Real Time package group.
# dnf groupinstall RTThis group installs several packages:
-
kernel-rtincludes meta-package that carries the list of package dependencies to have a functional kernel-rt installed. -
kernel-rt-coreincludes the core RHEL for Real Time kernel package. -
kernel-rt-develincludes the proper kernel binary, compiled withPREEMPT_RTenabled, the package also contains configuration and scripts to drive the kernel installation. -
kernel-rt-modules-coreincludes the essential kernel modules required for a functional kernel. kernel-rt-modules provides commonly used kernel modules, a complement to kernel-modules-core. -
kernel-rt-modules-extraincludes the RHEL for Real Time kernel extra modules package. -
realtime-setupsets up the basic environment required by RHEL for Real Time. -
rtevalevaluates system suitability for RHEL for Real Time. -
rteval-loadsprovides source code forrtevalloads. -
tuned-profiles-realtimeincludes the additionalTuneDprofiles targeted to real-time.
-
Optional: The
tunapackage contains a tool that helps tune the real-time kernel workload. It automates CPU isolation and thread affinity operations from the command line or the GUI. This package is available in the base RHEL 10 repository.# dnf install tunaNoteWhen the RHEL for Real Time kernel is installed, it automatically becomes the default kernel. The system uses this kernel on the next boot. You can also configure other existing kernels variants, such as,
kernel,kernel-debug, orkernel-rt-debugas the default boot kernel. For more information, see Configuring kernel-rt as the default boot kernel.
Verification
Check the installation location and verify that the components have been installed successfully.
# rpm -ql realtime-setup /etc/security/limits.d/realtime.conf /etc/sysconfig/realtime-setup /etc/udev/rules.d/99-rhel-rt.rules /usr/bin/realtime-setup /usr/bin/rt-setup-kdump /usr/bin/slub_cpu_partial_off /usr/lib/.build-id /usr/lib/.build-id/a4 /usr/lib/.build-id/a4/da77908aa4c6f048939f3267f1c552c456d117 /usr/lib/systemd/system/rt-entsk.service /usr/lib/systemd/system/rt-setup.service /usr/sbin/kernel-is-rt /usr/sbin/rt-entsk
1.3. Available RPM packages in the RHEL for Real Time repository
The RHEL for Real Time repository provides RPM packages, including the kernel-rt package, kernel test packages for real-time development and debugging, and debuginfo packages for code tracing.
The repository includes the following packages:
-
kernel-rtpackage, which is the RHEL for Real Time kernel package. - RHEL for Real Time kernel test packages, which contains test programs for the real-time kernel.
- RHEL for Real Time debuginfo packages, which are for debugging and code tracing.
| RPM package name | Description | RT-specific | Required |
|---|---|---|---|
|
| Low latency and preemption functionality | Yes | Yes |
| RPM package name | Description | RT-specific | Required |
|---|---|---|---|
|
| Headers and libraries for kernel development | Yes | No |
|
| RHEL for Real Time kernel with debugging functions compiled in (slow) | Yes | No |
|
| Headers and libraries for development on debug kernel | Yes | No |
|
| Utilities for measuring system latencies and for proving that priority-inheritance mutex functions properly | No | No |
The debuginfo packages are provided to use with the perf, trace-cmd, and crash utilities for analyzing kernel crash dumps. The debuginfo packages include symbol tables and are quite large. For this reason, the debuginfo packages are separately delivered from the other RHEL for Real Time packages. You can download the debugging packages with RHEL-RT as part of the RHEL offer.
| RPM package name | Description | RT-specific | Required |
|---|---|---|---|
|
|
Symbols for profiling and debugging use, such as | Yes | No |
|
| Symbols for profiling and tracing | Yes | No |
1.4. Post installation instructions
After you install the real-time kernel, perform system tuning with the provided tuned profiles. Configure kdump for crash dump information and verify that the Real Time kernel is the default kernel.
After installing the real-time kernel, ensure the following:
- To achieve optimal low-latency determinism, perform RHEL for Real Time specific system tuning by using the provided tuned profiles. Use these profiles as the final tuning or as the baseline for your specific tuning requirements.
- Review the module compatibility information for the real-time kernel and the standard kernel.
-
To enable
kdump, configure RHEL for Real Time to provide crash dump information by enablingkexec/kdump. - Verify that the Real Time kernel is the default kernel.
1.4.1. Module compatibility of the real-time kernel and the standard kernel
Kernel modules are inherently specific to the kernel they are built for. RHEL for Real Time and RHEL share a common code base. However, RHEL for Real Time has the real-time features and capabilities enabled. As a consequence, modules must be rebuilt to be used with RHEL for Real Time.
If you must use a third-party module, recompile it with the RHEL for Real Time header files. These files are available in the RHEL for Real Time development and test packages.
The following third-party drivers for standard Red Hat Enterprise Linux 10 do not currently have a RHEL for Real Time build:
- EMC Powerpath
- NVIDIA graphics
- Advanced storage adapter configuration utilities from Qlogic
User-space tools built for Red Hat Enterprise Linux 10 for the stock kernel are fully compatible with RHEL for Real Time. Some applications might not fully benefit from real-time optimizations due to their design. These applications might behave differently than expected.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}