Chapter 13. Using CPU Manager
13.1. What CPU Manager Does
CPU Manager manages groups of CPUs and constrains workloads to specific CPUs.
CPU Manager is useful for workloads that have some of these attributes:
- Require as much CPU time as possible.
- Are sensitive to processor cache misses.
- Are low-latency network applications.
- Coordinate with other processes and benefit from sharing a single processor cache.
13.2. Setting up CPU Manager
To set up CPU Manager:
Optionally, label a node:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc label node perf-node.example.com cpumanager=true
# oc label node perf-node.example.com cpumanager=true
Enable CPU manager support on the target node:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc edit configmap <name> -n openshift-node
# oc edit configmap <name> -n openshift-node
For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc edit cm node-config-compute -n openshift-node
# oc edit cm node-config-compute -n openshift-node
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ... kubeletArguments: ... feature-gates: - CPUManager=true cpu-manager-policy: - static cpu-manager-reconcile-period: - 5s system-reserved: - cpu=500m
... kubeletArguments: ... feature-gates: - CPUManager=true cpu-manager-policy: - static cpu-manager-reconcile-period: - 5s system-reserved:
1 - cpu=500m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl restart atomic-openshift-node
# systemctl restart atomic-openshift-node
- 1
system-reserved
is a required setting. The value might need to be adjusted depending on your environment.
Create a pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this pod:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat cpumanager.yaml
# cat cpumanager.yaml
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow apiVersion: v1 kind: Pod metadata: generateName: cpumanager- spec: containers: - name: cpumanager image: gcr.io/google_containers/pause-amd64:3.0 resources: requests: cpu: 1 memory: "1G" limits: cpu: 1 memory: "1G" nodeSelector: cpumanager: "true"
apiVersion: v1 kind: Pod metadata: generateName: cpumanager- spec: containers: - name: cpumanager image: gcr.io/google_containers/pause-amd64:3.0 resources: requests: cpu: 1 memory: "1G" limits: cpu: 1 memory: "1G" nodeSelector: cpumanager: "true"
Create the pod:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc create -f cpumanager.yaml
# oc create -f cpumanager.yaml
Verify that the pod is scheduled to the node that you labeled:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc describe pod cpumanager
# oc describe pod cpumanager
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Name: cpumanager-4gdtn Namespace: test Node: perf-node.example.com/172.31.62.105 ... Limits: cpu: 1 memory: 1G Requests: cpu: 1 memory: 1G ... QoS Class: Guaranteed Node-Selectors: cpumanager=true region=primary
Name: cpumanager-4gdtn Namespace: test Node: perf-node.example.com/172.31.62.105 ... Limits: cpu: 1 memory: 1G Requests: cpu: 1 memory: 1G ... QoS Class: Guaranteed Node-Selectors: cpumanager=true region=primary
Verify that the
cgroups
are set up correctly. Get the PID of the pause process:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemd-cgls -l
# systemd-cgls -l
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ├─1 /usr/lib/systemd/systemd --system --deserialize 20 ├─kubepods.slice │ ├─kubepods-pod0ec1ab8b_e1c4_11e7_bb22_027b30990a24.slice │ │ ├─docker-b24e29bc4021064057f941dc5f3538595c317d294f2c8e448b5e61a29c026d1c.scope │ │ │ └─44216 /pause
├─1 /usr/lib/systemd/systemd --system --deserialize 20 ├─kubepods.slice │ ├─kubepods-pod0ec1ab8b_e1c4_11e7_bb22_027b30990a24.slice │ │ ├─docker-b24e29bc4021064057f941dc5f3538595c317d294f2c8e448b5e61a29c026d1c.scope │ │ │ └─44216 /pause
Pods of QoS tier
Guaranteed
are placed within thekubepods.slice
. Pods of other QoS tiers end up in childcgroups
ofkubepods
.Copy to Clipboard Copied! Toggle word wrap Toggle overflow cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod0ec1ab8b_e1c4_11e7_bb22_027b30990a24.slice/docker-b24e29bc4021064057f941dc5f3538595c317d294f2c8e448b5e61a29c026d1c.scope for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done
# cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod0ec1ab8b_e1c4_11e7_bb22_027b30990a24.slice/docker-b24e29bc4021064057f941dc5f3538595c317d294f2c8e448b5e61a29c026d1c.scope # for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow cpuset.cpus 2 tasks 44216
cpuset.cpus 2 tasks 44216
Check the allowed CPU list for the task:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow grep ^Cpus_allowed_list /proc/44216/status
# grep ^Cpus_allowed_list /proc/44216/status
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Cpus_allowed_list: 2
Cpus_allowed_list: 2
Verify that another pod (in this case, the pod in the
burstable
QoS tier) on the system can not run on the core allocated for theGuaranteed
pod:Copy to Clipboard Copied! Toggle word wrap Toggle overflow cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe76ff22_dead_11e7_b99e_027b30990a24.slice/docker-da621bea7569704fc39f84385a179923309ab9d832f6360cccbff102e73f9557.scope/cpuset.cpus 0-1,3
# cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe76ff22_dead_11e7_b99e_027b30990a24.slice/docker-da621bea7569704fc39f84385a179923309ab9d832f6360cccbff102e73f9557.scope/cpuset.cpus 0-1,3
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc describe node perf-node.example.com
# oc describe node perf-node.example.com
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ... Capacity: cpu: 4 memory: 16266720Ki pods: 40 Allocatable: cpu: 3500m memory: 16164320Ki pods: 40 --- Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- test cpumanager-4gdtn 1 (28%) 1 (28%) 1G (6%) 1G (6%) test cpumanager-hczts 1 (28%) 1 (28%) 1G (6%) 1G (6%) test cpumanager-r9wrq 1 (28%) 1 (28%) 1G (6%) 1G (6%) ... Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 3 (85%) 3 (85%) 5437500k (32%) 9250M (55%)
... Capacity: cpu: 4 memory: 16266720Ki pods: 40 Allocatable: cpu: 3500m memory: 16164320Ki pods: 40 --- Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- test cpumanager-4gdtn 1 (28%) 1 (28%) 1G (6%) 1G (6%) test cpumanager-hczts 1 (28%) 1 (28%) 1G (6%) 1G (6%) test cpumanager-r9wrq 1 (28%) 1 (28%) 1G (6%) 1G (6%) ... Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 3 (85%) 3 (85%) 5437500k (32%) 9250M (55%)
This VM has four CPU cores. You set
system-reserved
to 500 millicores, meaning half of one core is subtracted from the total capacity of the node to arrive at theNode Allocatable
amount.You can see that
Allocatable CPU
is 3500 millicores. This means we can run three of our CPU manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores.If you try to schedule a fourth pod, the system will accept the pod, but it will never be scheduled:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get pods --all-namespaces |grep test
# oc get pods --all-namespaces |grep test
Example Output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow test cpumanager-4gdtn 1/1 Running 0 8m test cpumanager-hczts 1/1 Running 0 8m test cpumanager-nb9d5 0/1 Pending 0 8m test cpumanager-r9wrq 1/1 Running 0 8m
test cpumanager-4gdtn 1/1 Running 0 8m test cpumanager-hczts 1/1 Running 0 8m test cpumanager-nb9d5 0/1 Pending 0 8m test cpumanager-r9wrq 1/1 Running 0 8m