4.3. Per-group Division of CPU and Memory Resources
When a large amount of users use a single system, it is practical to provide certain users with more resources than others. Consider the following example: in a hypothetical company, there are three departments — finance, sales, and engineering. Because engineers use the system and its resources for more tasks than the other departments, it is logical that they have more resources available in case all departments are running CPU and memory intensive tasks.
Cgroups provide a way to limit the resources per each system group of users. For this example, assume that the following users have been created on the system:
~]$ grep home /etc/passwd
martin:x:500:500::/home/martin:/bin/bash
john:x:501:501::/home/john:/bin/bash
mark:x:502:502::/home/mark:/bin/bash
peter:x:503:503::/home/peter:/bin/bash
jenn:x:504:504::/home/jenn:/bin/bash
mike:x:505:505::/home/mike:/bin/bash
These users have been assigned to the following system groups:
~]$ grep -e "50[678]" /etc/group
finance:x:506:jenn,john
sales:x:507:mark,martin
engineering:x:508:peter,mike
For this example to work properly, you must have the libcgroup package installed. Using the
/etc/cgconfig.conf
and /etc/cgrules.conf
files, you can create a hierarchy and a set of rules which determine the amount of resources for each user. To achieve this, follow the steps in Procedure 4.3, “Per-group CPU and memory resource management”.
Procedure 4.3. Per-group CPU and memory resource management
- In the
/etc/cgconfig.conf
file, configure the following subsystems to be mounted and cgroups to be created:mount { cpu = /cgroup/cpu_and_mem; cpuacct = /cgroup/cpu_and_mem; memory = /cgroup/cpu_and_mem; } group finance { cpu { cpu.shares="250"; } cpuacct { cpuacct.usage="0"; } memory { memory.limit_in_bytes="2G"; memory.memsw.limit_in_bytes="3G"; } } group sales { cpu { cpu.shares="250"; } cpuacct { cpuacct.usage="0"; } memory { memory.limit_in_bytes="4G"; memory.memsw.limit_in_bytes="6G"; } } group engineering { cpu { cpu.shares="500"; } cpuacct { cpuacct.usage="0"; } memory { memory.limit_in_bytes="8G"; memory.memsw.limit_in_bytes="16G"; } }
When loaded, the above configuration file mounts thecpu
,cpuacct
, andmemory
subsystems to a singlecpu_and_mem
cgroup. For more information on these subsystems, refer to Chapter 3, Subsystems and Tunable Parameters. Next, it creates a hierarchy incpu_and_mem
which contains three cgroups: sales, finance, and engineering. In each of these cgroups, custom parameters are set for each subsystem:cpu
— thecpu.shares
parameter determines the share of CPU resources available to each process in all cgroups. Setting the parameter to250
,250
, and500
in the finance, sales, and engineering cgroups respectively means that processes started in these groups will split the resources with a 1:1:2 ratio. Note that when a single process is running, it consumes as much CPU as necessary no matter which cgroup it is placed in. The CPU limitation only comes into effect when two or more processes compete for CPU resources.cpuacct
— thecpuacct.usage="0"
parameter is used to reset values stored in thecpuacct.usage
andcpuacct.usage_percpu
files. These files report total CPU time (in nanoseconds) consumed by all processes in a cgroup.memory
— thememory.limit_in_bytes
parameter represents the amount of memory that is made available to all processes within a certain cgroup. In our example, processes started in the finance cgroup have 2 GB of memory available, processes in the sales group have 4 GB of memory available, and processes in the engineering group have 8 GB of memory available. Thememory.memsw.limit_in_bytes
parameter specifies the total amount of memory and swap space processes may use. Should a process in the finance cgroup hit the 2 GB memory limit, it is allowed to use another 1 GB of swap space, thus totaling the configured 3 GB.
- To define the rules which the
cgrulesengd
daemon uses to move processes to specific cgroups, configure the/etc/cgrules.conf
in the following way:#<user/group> <controller(s)> <cgroup> @finance cpu,cpuacct,memory finance @sales cpu,cpuacct,memory sales @engineering cpu,cpuacct,memory engineering
The above configuration creates rules that assign a specific system group (for example,@finance
) the resource controllers it may use (for example,cpu
,cpuacct
,memory
) and a cgroup (for example,finance
) which contains all processes originating from that system group.In our example, when thecgrulesengd
daemon, started via theservice cgred start
command, detects a process that is started by a user that belongs to the finance system group (for example,jenn
), that process is automatically moved to the/cgroup/cpu_and_mem/finance/tasks
file and is subjected to the resource limitations set in the finance cgroup. - Start the
cgconfig
service to create the hierarchy of cgroups and set the needed parameters in all created cgroups:~]#
service cgconfig start
Starting cgconfig service: [ OK ]Start thecgred
service to let thecgrulesengd
daemon detect any processes started in system groups configured in the/etc/cgrules.conf
file:~]#
service cgred start
Starting CGroup Rules Engine Daemon: [ OK ]Note thatcgred
is the name of the service that starts thecgrulesengd
daemon. - To make all of the changes above persistent across reboots, configure the
cgconfig
andcgred
services to be started by default:~]#
chkconfig cgconfig on
~]#chkconfig cgred on
To test whether this setup works, execute a CPU or memory intensive process and observe the results, for example, using the top utility. To test the CPU resource management, execute the following
dd
command under each user:
~]$ dd if=/dev/zero of=/dev/null bs=1024k
The above command reads the
/dev/zero
and outputs it to the /dev/null
in chunks of 1024 KB. When the top utility is launched, you can see results similar to these:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8201 peter 20 0 103m 1676 556 R 24.9 0.2 0:04.18 dd 8202 mike 20 0 103m 1672 556 R 24.9 0.2 0:03.47 dd 8199 jenn 20 0 103m 1676 556 R 12.6 0.2 0:02.87 dd 8200 john 20 0 103m 1676 556 R 12.6 0.2 0:02.20 dd 8197 martin 20 0 103m 1672 556 R 12.6 0.2 0:05.56 dd 8198 mark 20 0 103m 1672 556 R 12.3 0.2 0:04.28 dd ⋮
All processes have been correctly assigned to their cgroups and are only allowed to consume CPU resource made available to them. If all but two processes, which belong to the finance and engineering cgroups, are stopped, the remaining resources are evenly split between both processes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8202 mike 20 0 103m 1676 556 R 66.4 0.2 0:06.35 dd 8200 john 20 0 103m 1672 556 R 33.2 0.2 0:05.08 dd ⋮
Alternative method
Because the
cgrulesengd
daemon moves a process to a cgroup only after the appropriate conditions set by the rules in /etc/cgrules.conf
have been fulfilled, that process may be running for a few milliseconds in an incorrect cgroup. An alternative way to move processes to their specified cgroups is to use the pam_cgroup.so
PAM module. This module moves processes to available cgroups according to rules defined in the /etc/cgrules.conf
file. Follow the steps in Procedure 4.4, “Using a PAM module to move processes to cgroups” to configure the pam_cgroup.so
PAM module. Note that the libcgroup-pam package that provides this module is available form the Optional subscription channel. Before subscribing to this channel please see the Scope of Coverage Details. If you decide to install packages from the channel, follow the steps documented in the article called How to access Optional and Supplementary channels, and -devel packages using Red Hat Subscription Manager (RHSM)? on Red Hat Customer Portal.
Procedure 4.4. Using a PAM module to move processes to cgroups
- Install the libcgroup-pam package from the optional Red Hat Enterprise Linux Yum repository:
~]#
yum install libcgroup-pam --enablerepo=rhel-6-server-optional-rpms
- Ensure that the PAM module has been installed and exists:
~]#
ls /lib64/security/pam_cgroup.so
/lib64/security/pam_cgroup.soNote that on 32-bit systems, the module is placed in the/lib/security
directory. - Add the following line to the
/etc/pam.d/su
file to use thepam_cgroup.so
module each time thesu
command is executed:session optional pam_cgroup.so
- Configure the
/etc/cgconfig.conf
and/etc/cgrules.conf
files as in Procedure 4.3, “Per-group CPU and memory resource management”. - Log out all users that are affected by the cgroup settings in the
/etc/cgrules.conf
file to apply the above configuration.
When using the
pam_cgroup.so
PAM module, you may disable the cgred
service.