Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 14. Pruning objects to reclaim resources
Over time, API objects created in OpenShift Container Platform can accumulate in the cluster’s etcd data store through normal user operations, such as when building and deploying applications.
Cluster administrators can periodically prune older versions of objects from the cluster that are no longer required. For example, by pruning images you can delete older images and layers that are no longer in use, but are still taking up disk space.
14.1. Basic pruning operations Link kopierenLink in die Zwischenablage kopiert!
The CLI groups prune operations under a common parent command:
$ oc adm prune <object_type> <options>
This specifies:
-
The to perform the action on, such as
<object_type>,groups,builds, ordeployments.images -
The supported to prune that object type.
<options>
14.2. Pruning groups Link kopierenLink in die Zwischenablage kopiert!
To prune groups records from an external provider, administrators can run the following command:
$ oc adm prune groups \
--sync-config=path/to/sync/config [<options>]
| Options | Description |
|---|---|
|
| Indicate that pruning should occur, instead of performing a dry-run. |
|
| Path to the group blacklist file. |
|
| Path to the group whitelist file. |
|
| Path to the synchronization configuration file. |
Procedure
To see the groups that the prune command deletes, run the following command:
$ oc adm prune groups --sync-config=ldap-sync-config.yamlTo perform the prune operation, add the
flag:--confirm$ oc adm prune groups --sync-config=ldap-sync-config.yaml --confirm
14.3. Pruning deployment resources Link kopierenLink in die Zwischenablage kopiert!
You can prune resources associated with deployments that are no longer required by the system, due to age and status.
The following command prunes replication controllers associated with
DeploymentConfig
$ oc adm prune deployments [<options>]
To also prune replica sets associated with
Deployment
--replica-sets
| Option | Description |
|---|---|
|
| Indicate that pruning should occur, instead of performing a dry-run. |
|
| Per the
|
|
| Per the
|
|
| Do not prune any replication controller that is younger than
|
|
| Prune all replication controllers that no longer have a
|
|
| If
Important This flag is a Technology Preview feature. |
Procedure
To see what a pruning operation would delete, run the following command:
$ oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60mTo actually perform the prune operation, add the
flag:--confirm$ oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m --confirm
14.4. Pruning builds Link kopierenLink in die Zwischenablage kopiert!
To prune builds that are no longer required by the system due to age and status, administrators can run the following command:
$ oc adm prune builds [<options>]
| Option | Description |
|---|---|
|
| Indicate that pruning should occur, instead of performing a dry-run. |
|
| Prune all builds whose build configuration no longer exists, status is complete, failed, error, or canceled. |
|
| Per build configuration, keep the last
|
|
| Per build configuration, keep the last
|
|
| Do not prune any object that is younger than
|
Procedure
To see what a pruning operation would delete, run the following command:
$ oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60mTo actually perform the prune operation, add the
flag:--confirm$ oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m --confirm
Developers can enable automatic build pruning by modifying their build configuration.
14.5. Automatically pruning images Link kopierenLink in die Zwischenablage kopiert!
Images from the OpenShift image registry that are no longer required by the system due to age, status, or exceed limits are automatically pruned. Cluster administrators can configure the Pruning Custom Resource, or suspend it.
Prerequisites
- You have access to an OpenShift Container Platform cluster using an account with cluster administrator permissions.
-
Install the CLI.
oc
Procedure
-
Verify that the object named contains the following
imagepruners.imageregistry.operator.openshift.io/clusterandspecfields:status
spec:
schedule: 0 0 * * *
suspend: false
keepTagRevisions: 3
keepYoungerThanDuration: 60m
keepYoungerThan: 3600000000000
resources: {}
affinity: {}
nodeSelector: {}
tolerations: []
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
status:
observedGeneration: 2
conditions:
- type: Available
status: "True"
lastTransitionTime: 2019-10-09T03:13:45
reason: Ready
message: "Periodic image pruner has been created."
- type: Scheduled
status: "True"
lastTransitionTime: 2019-10-09T03:13:45
reason: Scheduled
message: "Image pruner job has been scheduled."
- type: Failed
staus: "False"
lastTransitionTime: 2019-10-09T03:13:45
reason: Succeeded
message: "Most recent image pruning job succeeded."
- 1
schedule:CronJobformatted schedule. This is an optional field, default is daily at midnight.- 2
suspend: If set totrue, theCronJobrunning pruning is suspended. This is an optional field, default isfalse. The initial value on new clusters isfalse.- 3
keepTagRevisions: The number of revisions per tag to keep. This is an optional field, default is3. The initial value is3.- 4
keepYoungerThanDuration: Retain images younger than this duration. This is an optional field. If a value is not specified, eitherkeepYoungerThanor the default value60m(60 minutes) is used.- 5
keepYoungerThan: Deprecated. The same askeepYoungerThanDuration, but the duration is specified as an integer in nanoseconds. This is an optional field. WhenkeepYoungerThanDurationis set, this field is ignored.- 6
resources: Standard pod resource requests and limits. This is an optional field.- 7
affinity: Standard pod affinity. This is an optional field.- 8
nodeSelector: Standard pod node selector. This is an optional field.- 9
tolerations: Standard pod tolerations. This is an optional field.- 10
successfulJobsHistoryLimit: The maximum number of successful jobs to retain. Must be>= 1to ensure metrics are reported. This is an optional field, default is3. The initial value is3.- 11
failedJobsHistoryLimit: The maximum number of failed jobs to retain. Must be>= 1to ensure metrics are reported. This is an optional field, default is3. The initial value is3.- 12
observedGeneration: The generation observed by the Operator.- 13
conditions: The standard condition objects with the following types:-
: Indicates if the pruning job has been created. Reasons can be Ready or Error.
Available -
: Indicates if the next pruning job has been scheduled. Reasons can be Scheduled, Suspended, or Error.
Scheduled -
: Indicates if the most recent pruning job failed.
Failed
-
The Image Registry Operator’s behavior for managing the pruner is orthogonal to the
managementState
ClusterOperator
Managed
However, the
managementState
-
: the
Managedflag for the image pruner is set to--prune-registry.true -
: the
Removedflag for the image pruner is set to--prune-registry, meaning it only prunes image metadata in etcd.false
14.6. Manually pruning images Link kopierenLink in die Zwischenablage kopiert!
The pruning custom resource enables automatic image pruning for the images from the OpenShift image registry. Administrators can manually prune images with the
oc adm prune images <image_prune_option>
$ oc adm prune images <image_prune_option>
- 1
- For more information about available pruning options, see "Manual image pruning command options".
This command removes images that are no longer required by the system.
Depending on your needs, you can prune images based on their age and tag history, or prune images that cause a project to exceed its defined storage limits.
14.6.1. Considerations when pruning images Link kopierenLink in die Zwischenablage kopiert!
Consider the following information before manually pruning images:
-
Pruning with the flag does not remove images. It only removes image streams, because images are cluster-scoped resources. Limiting pruning to a particular namespace makes it impossible to calculate current usage.
--namespace By default, the integrated registry caches metadata of blobs to reduce the number of requests to storage, and to increase request-processing speed. Pruning does not update the integrated registry cache. Images that still contain pruned layers after pruning will be broken because the pruned layers that have metadata in the cache will not be pushed. Therefore, you must redeploy the registry to clear the cache after pruning:
$ oc rollout restart deployment/image-registry -n openshift-image-registry- If the integrated registry uses a Redis cache, you must clean the database manually.
- If redeploying the registry after pruning is not an option, then you must permanently disable the cache.
-
operations require a route for your registry. Registry routes are not created by default.
oc adm prune images
14.6.2. Limitations when pruning images Link kopierenLink in die Zwischenablage kopiert!
The following limitations apply when pruning an image:
- Pruning images from external registries is unsupported.
-
When an image is pruned, all references to the image are removed from all image streams that contain the image in .
status.tags - Image layers that are no longer referenced by any images are removed.
14.6.3. Image prune conditions Link kopierenLink in die Zwischenablage kopiert!
OpenShift Container Platform supports two methodologies for pruning images:
- Pruning by age and tag
- Pruning by size limit
These methodologies are mutually exclusive. You must choose whether to prune by age and tag, or by size limit. Regardless of the method that you choose, the image pruner checks to ensure that images in use are not removed.
An image is only pruned if it meets the primary condition and is not actively referenced by a system component.
14.6.3.1. Pruning an image by age and tag Link kopierenLink in die Zwischenablage kopiert!
Pruning an image by age and tag is the default pruning strategy. It identifies images for removal by using the
--keep-younger-than
--keep-tag-revisions
--keep-younger-than
For an image to be pruned by age and tag, all of the following conditions must be met:
-
The image is managed by OpenShift Container Platform or has the annotation.
openshift.io/image.managed -
The image is older than the time specified by the flag.
--keep-younger-than -
The image is not one of the most recent images for its tag, as specified by the flag.
--keep-tag-revisions The image is not currently referenced by any of the following active or recent API objects:
-
Pods or image streams created more recently than the duration.
--keep-younger-than - Running or pending pods
- Deployments, replication controllers, replica sets, or stateful sets.
- Builds, build configurations, jobs, or cronjobs.
-
Pods or image streams created more recently than the
An image is only removed if it is old, not a recent tag revision, and is confirmed to have no active references by system components.
14.6.3.2. Pruning an image by size limit Link kopierenLink in die Zwischenablage kopiert!
Pruning an image by size limit uses the
--prune-over-size-limit
The
--prune-over-size-limit
--keep-tag-revisions
--keep-younger-than
For an image to be pruned using this method, all of the following conditions must be true:
- The image is part of a project that is currently exceeding its smallest defined size limit.
- The image is selected by the pruner as a candidate for deletion to reduce the total size.
The image is not currently referenced by any of the following active API objects:
-
Pods that are in a or
runningstate.pending - Deployments, replication controllers, replica sets, or stateful sets.
- Builds, build configurations, jobs, or cronjobs.
-
Pods that are in a
With this method, the primary trigger is the project’s size, but the safety check to ensure that the image is not actively in use is still performed.
14.6.4. Running image prune operations Link kopierenLink in die Zwischenablage kopiert!
Use the following procedure to run an image prune operation
Prerequisites
- You must be logged into the CLI with an access token.
-
You must have the cluster role or greater (for example,
system:image-pruner).cluster-admin - The image registry must be exposed.
- You have reviewed the "Considerations when manually pruning images" section of this document.
Procedure
Optional: To preview which images would be pruned, enter the following command. This command prints a list of the images, image streams, and pods that would be removed. Note that nothing is deleted until you add the
flag.--confirm$ oc adm prune images <image_prune_option_one> <image_prune_option_two>1 - 1
- For more information about available pruning options, see "Manual image pruning command options".
- Review the output to confirm the list of images, image streams, and pods to be removed.
Run the
command with the appropriate options for your cluster. Add theoc adm prune imagesflag to confirm deletion. For example:--confirm$ oc adm prune images <image_prune_option_one> <image_prune_option_two> --confirm
14.6.5. Using secure or insecure connections Link kopierenLink in die Zwischenablage kopiert!
The secure connection is the preferred and recommended approach. It is done over HTTPS protocol with a mandatory certificate verification. The
prune
The fall-back to insecure connection is allowed in the following cases unless
--certificate-authority
-
The command is run with the
pruneoption.--force-insecure -
The provided is prefixed with the
registry-urlscheme.http:// -
The provided is a local-link address or
registry-url.localhost -
The configuration of the current user allows for an insecure connection. This can be caused by the user either logging in using or choosing the insecure connection when prompted.
--insecure-skip-tls-verify
If the registry is secured by a certificate authority different from the one used by OpenShift Container Platform, it must be specified using the
--certificate-authority
prune
14.6.6. Image pruning CLI options Link kopierenLink in die Zwischenablage kopiert!
The following table describes the options you can use with the
oc adm prune images <image_prune_option>
| Option | Description |
|---|---|
|
| Include images that were not pushed to the registry, but have been mirrored by pullthrough. This is on by default. To limit the pruning to images that were pushed to the integrated registry, pass
|
|
| The path to a certificate authority file to use when communicating with the OpenShift Container Platform-managed registries. Defaults to the certificate authority data from the current user’s configuration file. If provided, a secure connection is initiated. |
|
| Indicate that pruning should occur, instead of performing a test-run. This requires a valid route to the integrated container image registry. If this command is run outside of the cluster network, the route must be provided using
|
|
| Use caution with this option. Allow an insecure connection to the container registry that is hosted via HTTP or has an invalid HTTPS certificate. |
|
| For each imagestream, keep up to at most
|
|
| Do not prune any image that is younger than
|
|
| Prune each image that exceeds the smallest limit defined in the same project. This flag cannot be combined with
|
|
| The address to use when contacting the registry. The command attempts to use a cluster-internal URL determined from managed images and image streams. In case it fails (the registry cannot be resolved or reached), an alternative route that works needs to be provided using this flag. The registry hostname can be prefixed by
|
|
| In conjunction with the conditions stipulated by the other options, this option controls whether the data in the registry corresponding to the OpenShift Container Platform image API object is pruned. By default, image pruning processes both the image API objects and corresponding data in the registry. This option is useful when you are only concerned with removing etcd content, to reduce the number of image objects but are not concerned with cleaning up registry storage, or if you intend to do that separately by hard pruning the registry during an appropriate maintenance window for the registry. |
14.6.6.1. Additional information about the --prune-registry flag Link kopierenLink in die Zwischenablage kopiert!
You can separate the removal of OpenShift Container Platform image API objects from the removal of image data in the registry by passing in the
--prune-registry=false
$ oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm --prune-registry=false
Then, you can perform a hard prune of the registry to remove the associated image data. This approach can narrow the timing window for race conditions compared to pruning both in a single command.
However, timing windows are not completely eliminated. For example, a pod might still be created that references an image while that image is being identified for pruning. You should track any API objects created during pruning to ensure that they do not reference deleted content.
Re-running the pruning without the
--prune-registry
--prune-registry=true
--prune-registry=false
14.6.7. Image pruning problems Link kopierenLink in die Zwischenablage kopiert!
14.6.7.1. Images not being pruned Link kopierenLink in die Zwischenablage kopiert!
If your images keep accumulating and the
prune
Ensure that images you want removed occur at higher positions in each tag history than your chosen tag revisions threshold. For example, consider an old and obsolete image named
sha256:abz
myapp
$ oc get is -n <namespace> -o go-template='{{range $isi, $is := .items}}{{range $ti, $tag := $is.status.tags}}'\
'{{range $ii, $item := $tag.items}}{{if eq $item.image "sha256:<hash>"}}{{$is.metadata.name}}:{{$tag.tag}} at position {{$ii}} out of {{len $tag.items}}\n'\
'{{end}}{{end}}{{end}}{{end}}'
Example output
myapp:v2 at position 4 out of 5
myapp:v2.1 at position 2 out of 2
myapp:v2.1-may-2016 at position 0 out of 1
When default options are used, the image is never pruned because it occurs at position
0
myapp:v2.1-may-2016
Specify
with the--keep-tag-revisions=0command.oc adm prune imagesWarningThis action removes all the tags from all the namespaces with underlying images, unless they are younger or they are referenced by objects younger than the specified threshold.
-
Delete all the where the position is below the revision threshold, which means
istagsandmyapp:v2.1.myapp:v2.1-may-2016 -
Move the image further in the history, either by running new builds pushing to the same , or by tagging other image. This is not always desirable for old release tags.
istag
Tags having a date or time of a particular image’s build in their names should be avoided, unless the image must be preserved for an undefined amount of time. Such tags tend to have just one image in their history, which prevents them from ever being pruned.
14.6.7.2. Using a secure connection against insecure registry Link kopierenLink in die Zwischenablage kopiert!
If you see a message similar to the following in the output of the
oc adm prune images
oc adm prune images
error: error communicating with registry: Get https://172.30.30.30:5000/healthz: http: server gave HTTP response to HTTPS client
-
The recommended solution is to secure the registry. Otherwise, you can force the client to use an insecure connection by appending to the command; however, this is not recommended.
--force-insecure
14.6.7.3. Using an insecure connection against a secured registry Link kopierenLink in die Zwischenablage kopiert!
If you see one of the following errors in the output of the
oc adm prune images
oc adm prune images
error: error communicating with registry: Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
error: error communicating with registry: [Get https://172.30.30.30:5000/healthz: x509: certificate signed by unknown authority, Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"]
By default, the certificate authority data stored in the user’s configuration files is used; the same is true for communication with the master API.
Use the
--certificate-authority
14.6.7.4. Using the wrong certificate authority Link kopierenLink in die Zwischenablage kopiert!
The following error means that the certificate authority used to sign the certificate of the secured container image registry is different from the authority used by the client:
error: error communicating with registry: Get https://172.30.30.30:5000/: x509: certificate signed by unknown authority
Make sure to provide the right one with the flag
--certificate-authority
As a workaround, the
--force-insecure
14.7. Hard pruning the registry Link kopierenLink in die Zwischenablage kopiert!
The OpenShift Container Registry can accumulate blobs that are not referenced by the OpenShift Container Platform cluster’s etcd. The basic pruning images procedure, therefore, is unable to operate on them. These are called orphaned blobs.
Orphaned blobs can occur from the following scenarios:
-
Manually deleting an image with command, which only removes the image from etcd, but not from the registry’s storage.
oc delete image <sha256:image-id> - Pushing to the registry initiated by daemon failures, which causes some blobs to get uploaded, but the image manifest (which is uploaded as the very last component) does not. All unique image blobs become orphans.
- OpenShift Container Platform refusing an image because of quota restrictions.
- The standard image pruner deleting an image manifest, but is interrupted before it deletes the related blobs.
- A bug in the registry pruner, which fails to remove the intended blobs, causing the image objects referencing them to be removed and the blobs becoming orphans.
Hard pruning the registry, a separate procedure from basic image pruning, allows cluster administrators to remove orphaned blobs. You should hard prune if you are running out of storage space in your OpenShift Container Registry and believe you have orphaned blobs.
This should be an infrequent operation and is necessary only when you have evidence that significant numbers of new orphans have been created. Otherwise, you can perform standard image pruning at regular intervals, for example, once a day (depending on the number of images being created).
Procedure
To hard prune orphaned blobs from the registry:
Log in.
Log in to the cluster with the CLI as
or another privileged user that has access to thekubeadminnamespace.openshift-image-registryRun a basic image prune.
Basic image pruning removes additional images that are no longer needed. The hard prune does not remove images on its own. It only removes blobs stored in the registry storage. Therefore, you should run this just before the hard prune.
Switch the registry to read-only mode.
If the registry is not running in read-only mode, any pushes happening at the same time as the prune will either:
- fail and cause new orphans, or
- succeed although the images cannot be pulled (because some of the referenced blobs were deleted).
Pushes will not succeed until the registry is switched back to read-write mode. Therefore, the hard prune must be carefully scheduled.
To switch the registry to read-only mode:
In
, setconfigs.imageregistry.operator.openshift.io/clustertospec.readOnly:true$ oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":true}}' --type=merge
Add the
system:image-prunerrole.The service account used to run the registry instances requires additional permissions to list some resources.
Get the service account name:
$ service_account=$(oc get -n openshift-image-registry \ -o jsonpath='{.spec.template.spec.serviceAccountName}' deploy/image-registry)Add the
cluster role to the service account:system:image-pruner$ oc adm policy add-cluster-role-to-user \ system:image-pruner -z \ ${service_account} -n openshift-image-registry
Optional: Run the pruner in dry-run mode.
To see how many blobs would be removed, run the hard pruner in dry-run mode. No changes are actually made. The following example references an image registry pod called
:image-registry-3-vhndw$ oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=check'Alternatively, to get the exact paths for the prune candidates, increase the logging level:
$ oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check'Example output
time="2017-06-22T11:50:25.066156047Z" level=info msg="start prune (dry-run mode)" distribution_version="v2.4.1+unknown" kubernetes_version=v1.6.1+$Format:%h$ openshift_version=unknown time="2017-06-22T11:50:25.092257421Z" level=info msg="Would delete blob: sha256:00043a2a5e384f6b59ab17e2c3d3a3d0a7de01b2cabeb606243e468acc663fa5" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092395621Z" level=info msg="Would delete blob: sha256:0022d49612807cb348cabc562c072ef34d756adfe0100a61952cbcb87ee6578a" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092492183Z" level=info msg="Would delete blob: sha256:0029dd4228961086707e53b881e25eba0564fa80033fbbb2e27847a28d16a37c" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.673946639Z" level=info msg="Would delete blob: sha256:ff7664dfc213d6cc60fd5c5f5bb00a7bf4a687e18e1df12d349a1d07b2cf7663" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674024531Z" level=info msg="Would delete blob: sha256:ff7a933178ccd931f4b5f40f9f19a65be5eeeec207e4fad2a5bafd28afbef57e" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674675469Z" level=info msg="Would delete blob: sha256:ff9b8956794b426cc80bb49a604a0b24a1553aae96b930c6919a6675db3d5e06" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 ... Would delete 13374 blobs Would free up 2.835 GiB of disk space Use -prune=delete to actually delete the dataRun the hard prune.
Execute the following command inside one running instance of a
pod to run the hard prune. The following example references an image registry pod calledimage-registry:image-registry-3-vhndw$ oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=delete'Example output
Deleted 13374 blobs Freed up 2.835 GiB of disk spaceSwitch the registry back to read-write mode.
After the prune is finished, the registry can be switched back to read-write mode. In
, setconfigs.imageregistry.operator.openshift.io/clustertospec.readOnly:false$ oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":false}}' --type=merge
14.8. Pruning cron jobs Link kopierenLink in die Zwischenablage kopiert!
Cron jobs can perform pruning of successful jobs, but might not properly handle failed jobs. Therefore, the cluster administrator should perform regular cleanup of jobs manually. They should also restrict the access to cron jobs to a small group of trusted users and set appropriate quota to prevent the cron job from creating too many jobs and pods.