Chapter 29. Using cgroupfs to manually manage cgroups

Procedure

1. Create the /sys/fs/cgroup/Example/ directory:

   # mkdir /sys/fs/cgroup/Example/

   The /sys/fs/cgroup/Example/ directory defines a child group. When you create the /sys/fs/cgroup/Example/ directory, some cgroups-v2 interface files are automatically created in the directory. The /sys/fs/cgroup/Example/ directory contains also controller-specific files for the memory and pids controllers.

2. Optional: Inspect the newly created child control group:

   # ll /sys/fs/cgroup/Example/

   -r--r--r--. 1 root root 0 Jun  1 10:33 cgroup.controllers
   -r--r--r--. 1 root root 0 Jun  1 10:33 cgroup.events
   -rw-r--r--. 1 root root 0 Jun  1 10:33 cgroup.freeze
   -rw-r--​r--. 1 root root 0 Jun  1 10:33 cgroup.procs
   ...
   -rw-r--r--. 1 root root 0 Jun  1 10:33 cgroup.subtree_control
   -r--r--r--. 1 root root 0 Jun  1 10:33 memory.events.local
   -rw-r--r--. 1 root root 0 Jun  1 10:33 memory.high
   -rw-r--r--. 1 root root 0 Jun  1 10:33 memory.low
   ...
   -r--r--r--. 1 root root 0 Jun  1 10:33 pids.current
   -r--r--r--. 1 root root 0 Jun  1 10:33 pids.events
   -rw-r--r--. 1 root root 0 Jun  1 10:33 pids.max

   The example output shows general cgroup control interface files such as cgroup.procs or cgroup.controllers. These files are common to all control groups, regardless of enabled controllers.

   The files such as memory.high and pids.max relate to the memory and pids controllers, which are in the root control group (/sys/fs/cgroup/), and are enabled by default by systemd.

   By default, the newly created child group inherits all settings from the parent cgroup. In this case, there are no limits from the root cgroup.

3. Verify that the required controllers are available in the /sys/fs/cgroup/cgroup.controllers file:

   # cat /sys/fs/cgroup/cgroup.controllers

   cpuset cpu io memory hugetlb pids rdma

4. Enable the required controllers. In this example it is cpu and cpuset controllers:

   # echo "+cpu" >> /sys/fs/cgroup/cgroup.subtree_control
   # echo "+cpuset" >> /sys/fs/cgroup/cgroup.subtree_control

   These commands enable the cpu and cpuset controllers for the immediate child groups of the /sys/fs/cgroup/ root control group. Including the newly created Example control group. A child group is where you can specify processes and apply control checks to each of the processes based on your criteria.

   Users can read the contents of the cgroup.subtree_control file at any level to get an idea of what controllers are available to enable in the immediate child group.

   Note

   By default, the /sys/fs/cgroup/cgroup.subtree_control file in the root control group contains memory and pids controllers.

5. Enable the required controllers for child cgroups of the Example control group:

   # echo "+cpu +cpuset" >> /sys/fs/cgroup/Example/cgroup.subtree_control

   This command ensures that the immediate child control group will only have controllers relevant to regulate the CPU time distribution - not to memory or pids controllers.

6. Create the /sys/fs/cgroup/Example/tasks/ directory:

   # mkdir /sys/fs/cgroup/Example/tasks/

   The /sys/fs/cgroup/Example/tasks/ directory defines a child group with files that relate purely to cpu and cpuset controllers. You can now assign processes to this control group and use cpu and cpuset controller options for your processes.

7. Optional: Inspect the child control group:

   # ll /sys/fs/cgroup/Example/tasks

   -r--r--r--. 1 root root 0 Jun  1 11:45 cgroup.controllers
   -r--r--r--. 1 root root 0 Jun  1 11:45 cgroup.events
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.freeze
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.max.depth
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.max.descendants
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.procs
   -r--r--r--. 1 root root 0 Jun  1 11:45 cgroup.stat
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.subtree_control
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.threads
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cgroup.type
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpu.max
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpu.pressure
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpuset.cpus
   -r--r--r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.effective
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.partition
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpuset.mems
   -r--r--r--. 1 root root 0 Jun  1 11:45 cpuset.mems.effective
   -r--r--r--. 1 root root 0 Jun  1 11:45 cpu.stat
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpu.weight
   -rw-r--r--. 1 root root 0 Jun  1 11:45 cpu.weight.nice
   -rw-r--r--. 1 root root 0 Jun  1 11:45 io.pressure
   -rw-r--r--. 1 root root 0 Jun  1 11:45 memory.pressure

   Important

   The cpu controller is only activated if the relevant child control group has at least 2 processes which compete for time on a single CPU.

Procedure

1. Configure the required CPU weights to achieve resource restrictions within the control groups:

   # echo "150" > /sys/fs/cgroup/Example/g1/cpu.weight
   # echo "100" > /sys/fs/cgroup/Example/g2/cpu.weight
   # echo "50" > /sys/fs/cgroup/Example/g3/cpu.weight

2. Add the applications' PIDs to the g1, g2, and g3 child groups:

   # echo "33373" > /sys/fs/cgroup/Example/g1/cgroup.procs
   # echo "33374" > /sys/fs/cgroup/Example/g2/cgroup.procs
   # echo "33377" > /sys/fs/cgroup/Example/g3/cgroup.procs

   These commands ensure that the required applications become members of the Example/g*/ child cgroups and will get their CPU time distributed based on the configuration of those cgroups.

   The weights of the children cgroups (g1, g2, g3) that have running processes are summed up at the level of the parent cgroup (Example). The CPU resource is then distributed proportionally based on the assigned weights.

   As a result, when all processes run at the same time, the kernel allocates to each of them the proportionate CPU time based on the assigned cgroup’s cpu.weight file:

   Expand

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~50% (150/300)
   g2	100	~33% (100/300)
   g3	50	~16% (50/300)

   Show more

   The value of the cpu.weight controller file is not a percentage.

   If one process stopped running, leaving cgroup g2 with no running processes, the calculation would omit the cgroup g2 and only account weights of cgroups g1 and g3:

   Expand

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~75% (150/200)
   g3	50	~25% (50/200)

   Show more
   Important

   If a child cgroup has multiple running processes, the CPU time allocated to the cgroup is distributed equally among its member processes.

Verification

1. Verify that the applications run in the specified control groups:

   # cat /proc/33373/cgroup /proc/33374/cgroup /proc/33377/cgroup

   0::/Example/g1
   0::/Example/g2
   0::/Example/g3

   The command output shows the processes of the specified applications that run in the Example/g*/ child cgroups.

2. Inspect the current CPU consumption of the throttled applications:

   # top

   top - 05:17:18 up 1 day, 18:25,  1 user,  load average: 3.03, 3.03, 3.00
   Tasks:  95 total,   4 running,  91 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 18.1 us, 81.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
   MiB Mem :   3737.0 total,   3233.7 free,    132.8 used,    370.5 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3373.1 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     33373 root      20   0   18720   1748   1460 R  *49.5*   0.0 415:05.87 sha1sum
     33374 root      20   0   18720   1756   1464 R  *32.9*   0.0 412:58.33 sha1sum
     33377 root      20   0   18720   1860   1568 R  *16.3*   0.0 411:03.12 sha1sum
       760 root      20   0  416620  28540  15296 S   0.3   0.7   0:10.23 tuned
         1 root      20   0  186328  14108   9484 S   0.0   0.4   0:02.00 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthread
   ...

   Note

   All processes run on a single CPU for clear illustration. The CPU weight applies the same principles when used on multiple CPUs.

   Notice that the CPU resource for the PID 33373, PID 33374, and PID 33377 was allocated based on the 150, 100, and 50 weights you assigned to child cgroups. The weights correspond to around 50%, 33%, and 16% allocation of CPU time for each application.