Chapter 23. Isolating CPUs using tuned-profiles-real-time
To give application threads the most execution time possible, you can isolate CPUs. Therefore, remove as many extraneous tasks from a CPU as possible. Isolating CPUs generally involves:
- Removing all user-space threads.
- Removing any unbound kernel threads. Kernel related bound threads are linked to a specific CPU and cannot not be moved).
-
Removing interrupts by modifying the
/proc/irq/N/smp_affinityproperty of each Interrupt Request (IRQ) numberNin the system.
By using the isolated_cores=cpulist configuration option of the tuned-profiles-realtime package, you can automate operations to isolate a CPU.
Prerequisites
- You have administrator privileges.
23.1. Choosing CPUs to isolate
Choosing the CPUs to isolate requires careful consideration of the CPU topology of the system. Different use cases require different configuration:
- If you have a multi-threaded application where threads need to communicate with one another by sharing cache, they need to be kept on the same NUMA node or physical socket.
- If you run multiple unrelated real-time applications, separating the CPUs by NUMA node or socket can be suitable.
The hwloc package provides utilities that are useful for getting information about CPUs, including lstopo-no-graphics and numactl.
Prerequisites
-
The
hwlocpackage are installed.
Procedure
View the layout of available CPUs in physical packages:
# lstopo-no-graphics --no-io --no-legend --of txtFigure 23.1. Showing the layout of CPUs using lstopo-no-graphics
This command is useful for multi-threaded applications, because it shows how many cores and sockets are available and the logical distance of the NUMA nodes.
Additionally, the
hwloc-guipackage provides thelstopoutility, which produces graphical output.View more information about the CPUs, such as the distance between nodes:
# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 node 0 size: 16159 MB node 0 free: 6323 MB node 1 cpus: 4 5 6 7 node 1 size: 16384 MB node 1 free: 10289 MB node distances: node 0 1 0: 10 21 1: 21 10
Additional resources
-
hwloc(7)man page on your system
23.2. Isolating CPUs using TuneD’s isolated_cores option
The initial mechanism for isolating CPUs is specifying the boot parameter isolcpus=cpulist on the kernel boot command line. The recommended way to do this for RHEL for Real Time is to use the TuneD daemon and its tuned-profiles-realtime package.
In tuned-profiles-realtime version 2.19 and later, the built-in function calc_isolated_cores applies the initial CPU setup automatically. The /etc/tuned/realtime-variables.conf configuration file includes the default variable content as isolated_cores=${f:calc_isolated_cores:2}.
By default, calc_isolated_cores reserves one core per socket for housekeeping and isolates the rest. If you must change the default configuration, comment out the isolated_cores=${f:calc_isolated_cores:2} line in /etc/tuned/realtime-variables.conf configuration file and follow the procedure steps for Isolating CPUs using TuneD’s isolated_cores option.
Prerequisites
-
The
TuneDandtuned-profiles-realtimepackages are installed. - You have root permissions on the system.
Procedure
-
As a root user, open
/etc/tuned/realtime-variables.confin a text editor. Set
isolated_cores=cpulistto specify the CPUs that you want to isolate. You can use CPU numbers and ranges.Examples:
isolated_cores=0-3,5,7
This isolates cores 0, 1, 2, 3, 5, and 7.
In a two socket system with 8 cores, where NUMA node 0 has cores 0-3 and NUMA node 1 has cores 4-8, to allocate two cores for a multi-threaded application, specify:
isolated_cores=4,5
This prevents any user-space threads from being assigned to CPUs 4 and 5.
To pick CPUs from different NUMA nodes for unrelated applications, specify:
isolated_cores=0,4
This prevents any user-space threads from being assigned to CPUs 0 and 4.
Activate the real-time
TuneDprofile using thetuned-admutility.# tuned-adm profile realtime- Reboot the machine for changes to take effect.
Verification
Search for the
isolcpusparameter in the kernel command line:$ cat /proc/cmdline | grep isolcpus BOOT_IMAGE=/vmlinuz-4.18.0-305.rt7.72.el8.x86_64 root=/dev/mapper/rhel_foo-root ro crashkernel=auto rd.lvm.lv=rhel_foo/root rd.lvm.lv=rhel_foo/swap console=ttyS0,115200n81 isolcpus=0,4
23.3. Isolating CPUs using the nohz and nohz_full parameters
The nohz and nohz_full parameters modify activity on specified CPUs. To enable these kernel boot parameters, you need to use one of the following TuneD profiles: realtime-virtual-host, realtime-virtual-guest, or cpu-partitioning.
nohz=onReduces timer activity on a particular set of CPUs.
The
nohzparameter is mainly used to reduce timer interrupts on idle CPUs. This helps battery life by allowing idle CPUs to run in reduced power mode. While not being directly useful for real-time response time, thenohzparameter does not directly impact real-time response time negatively. But thenohzparameter is required to activate thenohz_fullparameter that does have positive implications for real-time performance.nohz_full=cpulist-
The
nohz_fullparameter treats the timer ticks of a list of specified CPUs differently. If a CPU is specified as anohz_fullCPU and there is only one runnable task on the CPU, then the kernel stops sending timer ticks to that CPU. As a result, more time may be spent running the application and less time spent servicing interrupts and context switching.
Additional resources
