Red Hat SummitQuesto contenuto non è disponibile nella lingua selezionata.
Chapter 24. Using cgroups-v2 to control distribution of CPU time for applications
Prevent resource exhaustion by placing applications into control groups version 2 (cgroups-v2). By configuring CPU limits for these groups, you can regulate CPU consumption and ensure system stability.
The user has two methods how to regulate distribution of CPU time allocated to a control group:
-
Setting CPU bandwidth (editing the
cpu.maxcontroller file) -
Setting CPU weight (editing the
cpu.weightcontroller file)
24.1. Mounting cgroups-v2
RHEL 8 mounts cgroups-v1 by default. Configure the system manually to use cgroups-v2 for resource limiting. You can use systemd to control the resource usage. In special cases, you must manually configure cgroups, such as when you use cgroups-v1 controllers that have no cgroups-v2 equivalents.
Prerequisites
- You have root permissions.
Procedure
Configure the system to mount
cgroups-v2by default during system boot by the systemd system and service manager:# grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=1"This adds the necessary kernel command-line parameter to the current boot entry.
To add the
systemd.unified_cgroup_hierarchy=1parameter to all kernel boot entries:# grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"- Reboot the system for the changes to take effect.
Verification
Verify the
cgroups-v2filesystem is mounted:# mount -l | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)The
cgroups-v2filesystem was successfully mounted on the/sys/fs/cgroup/directory.Inspect the contents of the
/sys/fs/cgroup/directory:# ll /sys/fs/cgroup/ -r—r—r--. 1 root root 0 Apr 29 12:03 cgroup.controllers -rw-r—r--. 1 root root 0 Apr 29 12:03 cgroup.max.depth -rw-r—r--. 1 root root 0 Apr 29 12:03 cgroup.max.descendants -rw-r—r--. 1 root root 0 Apr 29 12:03 cgroup.procs -r—r—r--. 1 root root 0 Apr 29 12:03 cgroup.stat -rw-r—r--. 1 root root 0 Apr 29 12:18 cgroup.subtree_control -rw-r—r--. 1 root root 0 Apr 29 12:03 cgroup.threads -rw-r—r--. 1 root root 0 Apr 29 12:03 cpu.pressure -r—r—r--. 1 root root 0 Apr 29 12:03 cpuset.cpus.effective -r—r—r--. 1 root root 0 Apr 29 12:03 cpuset.mems.effective -r—r—r--. 1 root root 0 Apr 29 12:03 cpu.stat drwxr-xr-x. 2 root root 0 Apr 29 12:03 init.scope -rw-r—r--. 1 root root 0 Apr 29 12:03 io.pressure -r—r—r--. 1 root root 0 Apr 29 12:03 io.stat -rw-r—r--. 1 root root 0 Apr 29 12:03 memory.pressure -r—r—r--. 1 root root 0 Apr 29 12:03 memory.stat drwxr-xr-x. 69 root root 0 Apr 29 12:03 system.slice drwxr-xr-x. 3 root root 0 Apr 29 12:18 user.sliceThe
/sys/fs/cgroup/directory, also called the root control group, by default, provides interface files (starting withcgroup) and controller-specific files such ascpuset.cpus.effective. In addition, some directories related to systemd exist, such as,/sys/fs/cgroup/init.scope,/sys/fs/cgroup/system.slice, and/sys/fs/cgroup/user.slice.
24.2. Preparing the cgroup for distribution of CPU time
Enable CPU controllers and create dedicated control groups to manage application CPU consumption. For better organization, establish at least two levels of child control groups within the /sys/fs/cgroup/ directory.
Prerequisites
- You have root permissions.
- You have identified PIDs of processes that you want to control.
-
You have mounted the
cgroups-v2file system. For more information, see Mounting cgroups-v2.
Procedure
Identify the process IDs (PIDs) of applications whose CPU consumption you want to constrict:
# top Tasks: 104 total, 3 running, 101 sleeping, 0 stopped, 0 zombie %Cpu(s): 17.6 us, 81.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.8 hi, 0.0 si, 0.0 st MiB Mem : 3737.4 total, 3312.7 free, 133.3 used, 291.4 buff/cache MiB Swap: 4060.0 total, 4060.0 free, 0.0 used. 3376.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 34578 root 20 0 18720 1756 1468 R 99.0 0.0 0:31.09 sha1sum 34579 root 20 0 18720 1772 1480 R 99.0 0.0 0:30.54 sha1sum 1 root 20 0 186192 13940 9500 S 0.0 0.4 0:01.60 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp ...The example output reveals that
PID 34578and34579(two illustrative applications ofsha1sum) consume a huge amount of resources, namely CPU. Both are the example applications used to demonstrate managing thecgroups-v2functionality.Verify that the
cpuandcpusetcontrollers are available in the/sys/fs/cgroup/cgroup.controllersfile:# cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdmaEnable CPU-related controllers:
# echo "+cpu" >> /sys/fs/cgroup/cgroup.subtree_control # echo "+cpuset" >> /sys/fs/cgroup/cgroup.subtree_controlThese commands enable the
cpuandcpusetcontrollers for the immediate children groups of the/sys/fs/cgroup/root control group. A child group is where you can specify processes and apply control checks to each of the processes based on your criteria.You can review the
cgroup.subtree_controlfile at any level to identify the controllers that can be enabled in the immediate child group.NoteBy default, the
/sys/fs/cgroup/cgroup.subtree_controlfile in the root control group containsmemoryandpidscontrollers.Create the
/sys/fs/cgroup/Example/directory:# mkdir /sys/fs/cgroup/Example/The
/sys/fs/cgroup/Example/directory defines a child group. Also, the previous step enabled thecpuandcpusetcontrollers for this child group.When you create the
/sys/fs/cgroup/Example/directory, somecgroups-v2interface files andcpuandcpusetcontroller-specific files are automatically created in the directory. The/sys/fs/cgroup/Example/directory also provides controller-specific files for thememoryandpidscontrollers.Optional: Inspect the newly created child control group:
# ll /sys/fs/cgroup/Example/ -r—r—r--. 1 root root 0 Jun 1 10:33 cgroup.controllers -r—r—r--. 1 root root 0 Jun 1 10:33 cgroup.events -rw-r—r--. 1 root root 0 Jun 1 10:33 cgroup.freeze -rw-r—r--. 1 root root 0 Jun 1 10:33 cgroup.max.depth -rw-r—r--. 1 root root 0 Jun 1 10:33 cgroup.max.descendants -rw-r—r--. 1 root root 0 Jun 1 10:33 cgroup.procs -r—r—r--. 1 root root 0 Jun 1 10:33 cgroup.stat -rw-r—r--. 1 root root 0 Jun 1 10:33 cgroup.subtree_control … -rw-r—r--. 1 root root 0 Jun 1 10:33 cpuset.cpus -r—r—r--. 1 root root 0 Jun 1 10:33 cpuset.cpus.effective -rw-r—r--. 1 root root 0 Jun 1 10:33 cpuset.cpus.partition -rw-r—r--. 1 root root 0 Jun 1 10:33 cpuset.mems -r—r—r--. 1 root root 0 Jun 1 10:33 cpuset.mems.effective -r—r—r--. 1 root root 0 Jun 1 10:33 cpu.stat -rw-r—r--. 1 root root 0 Jun 1 10:33 cpu.weight -rw-r—r--. 1 root root 0 Jun 1 10:33 cpu.weight.nice … -r—r—r--. 1 root root 0 Jun 1 10:33 memory.events.local -rw-r—r--. 1 root root 0 Jun 1 10:33 memory.high -rw-r—r--. 1 root root 0 Jun 1 10:33 memory.low … -r—r—r--. 1 root root 0 Jun 1 10:33 pids.current -r—r—r--. 1 root root 0 Jun 1 10:33 pids.events -rw-r—r--. 1 root root 0 Jun 1 10:33 pids.maxThe example output shows files such as
cpuset.cpusandcpu.max. These files are specific to thecpusetandcpucontrollers. Thecpusetandcpucontrollers are manually enabled for the root’s (/sys/fs/cgroup/) direct child control groups using the/sys/fs/cgroup/cgroup.subtree_controlfile.The directory also includes general
cgroupcontrol interface files such ascgroup.procsorcgroup.controllers, which are common to all control groups, regardless of enabled controllers.The files such as
memory.highandpids.maxrelate to thememoryandpidscontrollers, which are in the root control group (/sys/fs/cgroup/), and are always enabled by default.By default, the newly created child group inherits access to all of the system’s CPU and memory resources, without any limits.
Enable the CPU-related controllers in
/sys/fs/cgroup/Example/to obtain controllers that are relevant only to CPU:# echo "+cpu" >> /sys/fs/cgroup/Example/cgroup.subtree_control # echo "+cpuset" >> /sys/fs/cgroup/Example/cgroup.subtree_controlThese commands ensure that the immediate child control group will only have controllers relevant to regulate the CPU time distribution - not to
memoryorpidscontrollers.Create the
/sys/fs/cgroup/Example/tasks/directory:# mkdir /sys/fs/cgroup/Example/tasks/The
/sys/fs/cgroup/Example/tasks/directory defines a child group with files that relate purely tocpuandcpusetcontrollers.Optional: Inspect another child control group:
# ll /sys/fs/cgroup/Example/tasks -r—r—r--. 1 root root 0 Jun 1 11:45 cgroup.controllers -r—r—r--. 1 root root 0 Jun 1 11:45 cgroup.events -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.freeze -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.max.depth -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.max.descendants -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.procs -r—r—r--. 1 root root 0 Jun 1 11:45 cgroup.stat -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.subtree_control -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.threads -rw-r—r--. 1 root root 0 Jun 1 11:45 cgroup.type -rw-r—r--. 1 root root 0 Jun 1 11:45 cpu.max -rw-r—r--. 1 root root 0 Jun 1 11:45 cpu.pressure -rw-r—r--. 1 root root 0 Jun 1 11:45 cpuset.cpus -r—r—r--. 1 root root 0 Jun 1 11:45 cpuset.cpus.effective -rw-r—r--. 1 root root 0 Jun 1 11:45 cpuset.cpus.partition -rw-r—r--. 1 root root 0 Jun 1 11:45 cpuset.mems -r—r—r--. 1 root root 0 Jun 1 11:45 cpuset.mems.effective -r—r—r--. 1 root root 0 Jun 1 11:45 cpu.stat -rw-r—r--. 1 root root 0 Jun 1 11:45 cpu.weight -rw-r—r--. 1 root root 0 Jun 1 11:45 cpu.weight.nice -rw-r—r--. 1 root root 0 Jun 1 11:45 io.pressure -rw-r—r--. 1 root root 0 Jun 1 11:45 memory.pressureEnsure the processes that you want to control for CPU time compete on the same CPU:
# echo "1" > /sys/fs/cgroup/Example/tasks/cpuset.cpusThis ensures the processes you will place in the
Example/taskschild control group, compete on the same CPU. This setting is important for thecpucontroller to activate.ImportantThe
cpucontroller is only activated if the relevant child control group has at least 2 processes to compete for time on a single CPU.
Verification
Optional: Ensure the CPU-related controllers are enabled for the immediate children cgroups:
# cat /sys/fs/cgroup/cgroup.subtree_control /sys/fs/cgroup/Example/cgroup.subtree_control cpuset cpu memory pids cpuset cpuOptional: Ensure the processes that you want to control for CPU time compete on the same CPU:
# cat /sys/fs/cgroup/Example/tasks/cpuset.cpus 1
24.3. Controlling distribution of CPU time for applications by adjusting CPU bandwidth
You need to assign values to the relevant files of the cpu controller to regulate distribution of the CPU time to applications under the specific cgroup tree.
Prerequisites
- You have root permissions.
- You have at least two applications for which you want to control distribution of CPU time.
- You ensured the relevant applications compete for CPU time on the same CPU as described in Preparing the cgroup for distribution of CPU time.
-
You mounted
cgroups-v2filesystem as described in Mounting cgroups-v2. -
You enabled
cpuandcpusetcontrollers both in the parent control group and in child control group similarly as described in Preparing the cgroup for distribution of CPU time. You created two levels of child control groups inside the
/sys/fs/cgroup/root control group as in the example below:… ├── Example │ ├── tasks …
Procedure
Configure CPU bandwidth to achieve resource restrictions within the control group:
# echo "200000 1000000" > /sys/fs/cgroup/Example/tasks/cpu.maxThe first value is the allowed time quota in microseconds for which all processes collectively in a child group can run during one period. The second value specifies the length of the period.
During a single period, when processes in a control group collectively exhaust the time specified by this quota, they are throttled for the remainder of the period and not allowed to run until the next period.
This command sets CPU time distribution controls so that all processes collectively in the
/sys/fs/cgroup/Example/taskschild group can run on the CPU for only 0.2 seconds of every 1 second. That is, one fifth of each second.Optional: Verify the time quotas:
# cat /sys/fs/cgroup/Example/tasks/cpu.max 200000 1000000Add the applications' PIDs to the
Example/taskschild group:# echo "34578" > /sys/fs/cgroup/Example/tasks/cgroup.procs # echo "34579" > /sys/fs/cgroup/Example/tasks/cgroup.procsThe example commands ensure that required applications become members of the
Example/taskschild group and do not exceed the CPU time distribution configured for this child group.
Verification
Verify that the applications run in the specified control group:
# cat /proc/34578/cgroup /proc/34579/cgroup 0::/Example/tasks 0::/Example/tasksThe output above shows the processes of the specified applications that run in the
Example/taskschild group.Inspect the current CPU consumption of the throttled applications:
# top top - 11:13:53 up 23:10, 1 user, load average: 0.26, 1.33, 1.66 Tasks: 104 total, 3 running, 101 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.0 us, 7.0 sy, 0.0 ni, 89.5 id, 0.0 wa, 0.2 hi, 0.2 si, 0.2 st MiB Mem : 3737.4 total, 3312.6 free, 133.4 used, 291.4 buff/cache MiB Swap: 4060.0 total, 4060.0 free, 0.0 used. 3376.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 34578 root 20 0 18720 1756 1468 R 10.0 0.0 37:36.13 sha1sum 34579 root 20 0 18720 1772 1480 R 10.0 0.0 37:41.22 sha1sum 1 root 20 0 186192 13940 9500 S 0.0 0.4 0:01.60 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp ...Notice that the CPU consumption for the
PID 34578andPID 34579has decreased to 10%. TheExample/taskschild group regulates its processes to 20% of the CPU time collectively. Since the control group contains 2 processes, each can use 10% of the CPU time.
24.4. Controlling distribution of CPU time for applications by adjusting CPU weight
You need to assign values to the relevant files of the cpu controller to regulate distribution of the CPU time to applications under the specific cgroup tree.
Prerequisites
- You have root permissions.
- You have applications for which you want to control distribution of CPU time.
- You ensured the relevant applications compete for CPU time on the same CPU as described in Preparing the cgroup for distribution of CPU time.
-
You mounted
cgroups-v2filesystem as described in Mounting cgroups-v2. You created a two level hierarchy of child control groups inside the
/sys/fs/cgroup/root control group as in the following example:… ├── Example │ ├── g1 │ ├── g2 │ └── g3 …-
You enabled
cpuandcpusetcontrollers in the parent control group and in child control groups similarly as described in Preparing the cgroup for distribution of CPU time.
Procedure
Configure desired CPU weights to achieve resource restrictions within the control groups:
# echo "150" > /sys/fs/cgroup/Example/g1/cpu.weight # echo "100" > /sys/fs/cgroup/Example/g2/cpu.weight # echo "50" > /sys/fs/cgroup/Example/g3/cpu.weightAdd the applications' PIDs to the
g1,g2, andg3child groups:# echo "33373" > /sys/fs/cgroup/Example/g1/cgroup.procs # echo "33374" > /sys/fs/cgroup/Example/g2/cgroup.procs # echo "33377" > /sys/fs/cgroup/Example/g3/cgroup.procsThe example commands ensure that desired applications become members of the
Example/g*/child cgroups and will get their CPU time distributed as per the configuration of those cgroups.The weights of the children cgroups (
g1,g2,g3) that have running processes are summed up at the level of the parent cgroup (Example). The CPU resource is then distributed proportionally based on their weights.As a result, when all processes run at the same time, the kernel allocates to each of them the proportionate CPU time based on their cgroup’s
cpu.weightfile:Expand Child cgroup cpu.weightfileCPU time allocation g1
150
~50% (150/300)
g2
100
~33% (100/300)
g3
50
~16% (50/300)
The value of the
cpu.weightcontroller file is not a percentage.If one process stopped running, leaving cgroup
g2with no running processes, the calculation would omit the cgroupg2and only account weights of cgroupsg1andg3:Expand Child cgroup cpu.weightfileCPU time allocation g1
150
~75% (150/200)
g3
50
~25% (50/200)
ImportantIf a child cgroup has multiple running processes, the CPU time allocated to the cgroup is distributed equally among its member processes.
Verification
Verify that the applications run in the specified control groups:
# cat /proc/33373/cgroup /proc/33374/cgroup /proc/33377/cgroup 0::/Example/g1 0::/Example/g2 0::/Example/g3The command output shows the processes of the specified applications that run in the
Example/g*/child cgroups.Inspect the current CPU consumption of the throttled applications:
# top top - 05:17:18 up 1 day, 18:25, 1 user, load average: 3.03, 3.03, 3.00 Tasks: 95 total, 4 running, 91 sleeping, 0 stopped, 0 zombie %Cpu(s): 18.1 us, 81.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st MiB Mem : 3737.0 total, 3233.7 free, 132.8 used, 370.5 buff/cache MiB Swap: 4060.0 total, 4060.0 free, 0.0 used. 3373.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 33373 root 20 0 18720 1748 1460 R 49.5 0.0 415:05.87 sha1sum 33374 root 20 0 18720 1756 1464 R 32.9 0.0 412:58.33 sha1sum 33377 root 20 0 18720 1860 1568 R 16.3 0.0 411:03.12 sha1sum 760 root 20 0 416620 28540 15296 S 0.3 0.7 0:10.23 tuned 1 root 20 0 186328 14108 9484 S 0.0 0.4 0:02.00 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthread ...NoteAll processes run on a single CPU for clear illustration. The CPU weight applies the same principles when used on multiple CPUs.
Notice that the CPU resource for the
PID 33373,PID 33374, andPID 33377was allocated based on the 150, 100, and 50 weights you assigned to the child cgroups. The weights correspond to around 50%, 33%, and 16% allocation of CPU time for each application.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}