2.13. Freeing node resources using garbage collection
Understand and use garbage collection.
2.13.1. Understanding how terminated containers are removed though garbage collection
Container garbage collection can be performed using eviction thresholds.
When eviction thresholds are set for garbage collection, the node tries to keep any container for any pod accessible from the API. If the pod has been deleted, the containers will be as well. Containers are preserved as long the pod is not deleted and the eviction threshold is not reached. If the node is under disk pressure, it will remove containers and their logs will no longer be accessible using oc logs.
- eviction-soft - A soft eviction threshold pairs an eviction threshold with a required administrator-specified grace period.
- eviction-hard - A hard eviction threshold has no grace period, and if observed, OpenShift Container Platform takes immediate action.
If a node is oscillating above and below a soft eviction threshold, but not exceeding its associated grace period, the corresponding node would constantly oscillate between true and false. As a consequence, the scheduler could make poor scheduling decisions.
To protect against this oscillation, use the eviction-pressure-transition-period flag to control how long OpenShift Container Platform must wait before transitioning out of a pressure condition. OpenShift Container Platform will not set an eviction threshold as being met for the specified pressure condition for the period specified before toggling the condition back to false.
2.13.2. Understanding how images are removed though garbage collection
Image garbage collection relies on disk usage as reported by cAdvisor on the node to decide which images to remove from the node.
The policy for image garbage collection is based on two conditions:
- The percent of disk usage (expressed as an integer) which triggers image garbage collection. The default is 85.
- The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free. Default is 80.
For image garbage collection, you can modify any of the following variables using a custom resource.
| Setting | Description |
|---|---|
|
| The minimum age for an unused image before the image is removed by garbage collection. The default is 2m. |
|
| The percent of disk usage, expressed as an integer, which triggers image garbage collection. The default is 85. |
|
| The percent of disk usage, expressed as an integer, to which image garbage collection attempts to free. The default is 80. |
Two lists of images are retrieved in each garbage collector run:
- A list of images currently running in at least one pod.
- A list of images available on a host.
As new containers are run, new images appear. All images are marked with a time stamp. If the image is running (the first list above) or is newly detected (the second list above), it is marked with the current time. The remaining images are already marked from the previous spins. All images are then sorted by the time stamp.
Once the collection starts, the oldest images get deleted first until the stopping criterion is met.
2.13.3. Configuring garbage collection for containers and images
As an administrator, you can configure how OpenShift Container Platform performs garbage collection by creating a kubeletConfig object for each machine config pool.
OpenShift Container Platform supports only one kubeletConfig object for each machine config pool.
You can configure any combination of the following:
- soft eviction for containers
- hard eviction for containers
- eviction for images
For soft container eviction you can also configure a grace period before eviction.
Prerequisites
Obtain the label associated with the static
MachineConfigPoolCRD for the type of node you want to configure. Perform one of the following steps:View the machine config pool:
$ oc describe machineconfigpool <name>
For example:
$ oc describe machineconfigpool worker
Example output
Name: worker Namespace: Labels: custom-kubelet=small-pods 1- 1
- If a label has been added it appears under
Labels.
If the label is not present, add a key/value pair:
$ oc label machineconfigpool worker custom-kubelet=small-pods
Procedure
Create a custom resource (CR) for your configuration change.
Sample configuration for a container garbage collection CR:
apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: worker-kubeconfig 1 spec: machineConfigPoolSelector: matchLabels: custom-kubelet: small-pods 2 kubeletConfig: evictionSoft: 3 memory.available: "500Mi" 4 nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%" imagefs.inodesFree: "10%" evictionSoftGracePeriod: 5 memory.available: "1m30s" nodefs.available: "1m30s" nodefs.inodesFree: "1m30s" imagefs.available: "1m30s" imagefs.inodesFree: "1m30s" evictionHard: memory.available: "200Mi" nodefs.available: "5%" nodefs.inodesFree: "4%" imagefs.available: "10%" imagefs.inodesFree: "5%" evictionPressureTransitionPeriod: 0s 6 imageMinimumGCAge: 5m 7 imageGCHighThresholdPercent: 80 8 imageGCLowThresholdPercent: 75 9
- 1
- Name for the object.
- 2
- Selector label.
- 3
- Type of eviction:
EvictionSoftandEvictionHard. - 4
- Eviction thresholds based on a specific eviction trigger signal.
- 5
- Grace periods for the soft eviction. This parameter does not apply to
eviction-hard. - 6
- The duration to wait before transitioning out of an eviction pressure condition
- 7
- The minimum age for an unused image before the image is removed by garbage collection.
- 8
- The percent of disk usage (expressed as an integer) which triggers image garbage collection.
- 9
- The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free.
Create the object:
$ oc create -f <file-name>.yaml
For example:
$ oc create -f gc-container.yaml
Example output
kubeletconfig.machineconfiguration.openshift.io/gc-container created
Verify that garbage collection is active. The Machine Config Pool you specified in the custom resource appears with
UPDATINGas 'true` until the change is fully implemented:$ oc get machineconfigpool
Example output
NAME CONFIG UPDATED UPDATING master rendered-master-546383f80705bd5aeaba93 True False worker rendered-worker-b4c51bb33ccaae6fc4a6a5 False True
