Chapter 7. Using Topology Manager
Topology Manager is a Kubelet component that collects hints from CPU Manager and Device Manager to align pod CPU and device resources on the same non-uniform memory access (NUMA) node.
Topology Manager uses topology information from collected hints to decide if the pod can be accepted or rejected from the node, based on the configured Topology Manager policy and Pod resources requested.
Topology Manager is useful for workloads that desire to use hardware accelerators to support latency-critical execution and high throughput parallel computation.
Topology Manager is an alpha feature in OpenShift Container Platform.
7.1. Setting up Topology Manager
Prequisites
-
Configure the CPU Manager policy to be
static. Refer to Using CPU Manager in the Scalability and Performance section.
Procedure
Enable the LatencySensitive FeatureGate
# oc edit featuregate/cluster
Add Feature Set: LatencySensitive to the spec:
# oc describe featuregate/cluster Name: cluster Namespace: Labels: <none> Annotations: release.openshift.io/create-only: true API Version: config.openshift.io/v1 Kind: FeatureGate Metadata: Creation Timestamp: 2019-10-30T15:06:41Z Generation: 2 Resource Version: 7773803 Self Link: /apis/config.openshift.io/v1/featuregates/cluster UID: b00204ab-cc5e-4ca5-ad93-b9bdd738c1de Spec: Feature Set: LatencySensitive Events: <none>
Configure the Topology Manager policy with KubeletConfig.
The example YAML file below shows a
single-numa-nodepolicy specified.apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: cpumanager-enabled spec: machineConfigPoolSelector: matchLabels: custom-kubelet: cpumanager-enabled kubeletConfig: cpuManagerPolicy: static cpuManagerReconcilePeriod: 5s topologyManagerPolicy: single-numa-node 1- 1
- Specify your selected Topology Manager policy.
# oc create -f topologymanager-kubeletconfig.yaml
7.2. Topology Manager policies
Topology Manager works on Nodes and pods that meet the following conditions:
-
The Node’s CPU Manager Policy is configured as
static. -
The pods are in the
GuaranteedQoS class.
When the above conditions are met, Topology Manager will align CPU and device requests for the Pod.
Topology Manager supports 4 allocation policies. These policies are set via a Kubelet flag, --topology-manager-policy. The policies are:
-
none(default) -
best-effort -
restricted -
single-numa-node
7.2.1. none policy
This is the default policy and does not perform any topology alignment.
7.2.2. best-effort policy
For each container in a pod with the best-effort topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, Topology Manager stores this and admits the pod to the node.
7.2.3. restricted policy
For each container in a pod with the restricted topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager stores the preferred NUMA Node affinity for that container. If the affinity is not preferred, Topology Manager rejects this pod from the node, resulting in a pod in a Terminated state with a pod admission failure.
7.2.4. single-numa-node
For each container in a pod with the single-numa-node topology management policy, kubelet calls each Hint Provider to discover their resource availability. Using this information, the Topology Manager determines if a single NUMA Node affinity is possible. If it is, the pod is admitted to the node. If a single NUMA Node affinity is not possible, the Topology Manager rejects the pod from the node. This results in a pod in a Terminated state with a pod admission failure.
7.3. Pod interactions with Topology Manager policies
The example Pod specs below help illustrate pod interactions with Topology Manager.
The following pod runs in the BestEffort QoS class because no resource requests or limits are specified.
spec:
containers:
- name: nginx
image: nginx
The next pod runs in the Burstable QoS class because requests are less than limits.
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
If the selected policy is anything other than none, Topology Manager would not consider either of these Pod specifications.
The last example pod below runs in the Guaranteed QoS class because requests are equal to limits.
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
example.com/device: "1"
requests:
memory: "200Mi"
cpu: "2"
example.com/device: "1"Topology Manager would consider this pod. The Topology Manager consults the CPU Manager static policy, which returns the topology of available CPUs. Topology Manager also consults Device Manager to discover the topology of available devices for example.com/device.
Topology Manager will use this information to store the best Topology for this container. In the case of this pod, CPU Manager and Device Manager will use this stored information at the resource allocation stage.
