Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 6. Deleting a machine
You can delete a specific machine.
6.1. Deleting a specific machine
You can delete a specific machine.
Do not delete a control plane machine unless your cluster uses a control plane machine set.
Prerequisites
- Install an OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc). -
Log in to
ocas a user withcluster-adminpermission.
Procedure
View the machines that are in the cluster by running the following command:
$ oc get machine -n openshift-machine-api
The command output contains a list of machines in the
<clusterid>-<role>-<cloud_region>format.- Identify the machine that you want to delete.
Delete the machine by running the following command:
$ oc delete machine <machine> -n openshift-machine-api
ImportantBy default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed. If the drain operation fails, the machine controller cannot proceed removing the machine.
You can skip draining the node by annotating
machine.openshift.io/exclude-node-drainingin a specific machine.If the machine that you delete belongs to a machine set, a new machine is immediately created to satisfy the specified number of replicas.
6.2. Lifecycle hooks for the machine deletion phase
Machine lifecycle hooks are points in the reconciliation lifecycle of a machine where the normal lifecycle process can be interrupted. In the machine Deleting phase, these interruptions provide the opportunity for components to modify the machine deletion process.
6.2.1. Terminology and definitions
To understand the behavior of lifecycle hooks for the machine deletion phase, you must understand the following concepts:
- Reconciliation
- Reconciliation is the process by which a controller attempts to make the real state of the cluster and the objects that it comprises match the requirements in an object specification.
- Machine controller
The machine controller manages the reconciliation lifecycle for a machine. For machines on cloud platforms, the machine controller is the combination of an OpenShift Container Platform controller and a platform-specific actuator from the cloud provider.
In the context of machine deletion, the machine controller performs the following actions:
- Drain the node that is backed by the machine.
- Delete the machine instance from the cloud provider.
-
Delete the
Nodeobject.
- Lifecycle hook
A lifecycle hook is a defined point in the reconciliation lifecycle of an object where the normal lifecycle process can be interrupted. Components can use a lifecycle hook to inject changes into the process to accomplish a desired outcome.
There are two lifecycle hooks in the machine
Deletingphase:-
preDrainlifecycle hooks must be resolved before the node that is backed by the machine can be drained. -
preTerminatelifecycle hooks must be resolved before the instance can be removed from the infrastructure provider.
-
- Hook-implementing controller
A hook-implementing controller is a controller, other than the machine controller, that can interact with a lifecycle hook. A hook-implementing controller can do one or more of the following actions:
- Add a lifecycle hook.
- Respond to a lifecycle hook.
- Remove a lifecycle hook.
Each lifecycle hook has a single hook-implementing controller, but a hook-implementing controller can manage one or more hooks.
6.2.2. Machine deletion processing order
In OpenShift Container Platform 4.17, there are two lifecycle hooks for the machine deletion phase: preDrain and preTerminate. When all hooks for a given lifecycle point are removed, reconciliation continues as normal.
Figure 6.1. Machine deletion flow
The machine Deleting phase proceeds in the following order:
An existing machine is slated for deletion for one of the following reasons:
-
A user with
cluster-adminpermissions uses theoc delete machinecommand. -
The machine gets a
machine.openshift.io/delete-machineannotation. - The machine set that manages the machine marks it for deletion to reduce the replica count as part of reconciliation.
- The cluster autoscaler identifies a node that is unnecessary to meet the deployment needs of the cluster.
- A machine health check is configured to replace an unhealthy machine.
-
A user with
-
The machine enters the
Deletingphase, in which it is marked for deletion but is still present in the API. If a
preDrainlifecycle hook exists, the hook-implementing controller that manages it does a specified action.Until all
preDrainlifecycle hooks are satisfied, the machine status conditionDrainableis set toFalse.-
There are no unresolved
preDrainlifecycle hooks and the machine status conditionDrainableis set toTrue. The machine controller attempts to drain the node that is backed by the machine.
-
If draining fails,
Drainedis set toFalseand the machine controller attempts to drain the node again. -
If draining succeeds,
Drainedis set toTrue.
-
If draining fails,
-
The machine status condition
Drainedis set toTrue. If a
preTerminatelifecycle hook exists, the hook-implementing controller that manages it does a specified action.Until all
preTerminatelifecycle hooks are satisfied, the machine status conditionTerminableis set toFalse.-
There are no unresolved
preTerminatelifecycle hooks and the machine status conditionTerminableis set toTrue. - The machine controller removes the instance from the infrastructure provider.
-
The machine controller deletes the
Nodeobject.
6.2.3. Deletion lifecycle hook configuration
The following YAML snippets demonstrate the format and placement of deletion lifecycle hook configurations within a machine set:
YAML snippet demonstrating a preDrain lifecycle hook
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
...
spec:
lifecycleHooks:
preDrain:
- name: <hook_name> 1
owner: <hook_owner> 2
...
YAML snippet demonstrating a preTerminate lifecycle hook
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
...
spec:
lifecycleHooks:
preTerminate:
- name: <hook_name> 1
owner: <hook_owner> 2
...
Example lifecycle hook configuration
The following example demonstrates the implementation of multiple fictional lifecycle hooks that interrupt the machine deletion process:
Example configuration for lifecycle hooks
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
...
spec:
lifecycleHooks:
preDrain: 1
- name: MigrateImportantApp
owner: my-app-migration-controller
preTerminate: 2
- name: BackupFileSystem
owner: my-backup-controller
- name: CloudProviderSpecialCase
owner: my-custom-storage-detach-controller 3
- name: WaitForStorageDetach
owner: my-custom-storage-detach-controller
...
6.2.4. Machine deletion lifecycle hook examples for Operator developers
Operators can use lifecycle hooks for the machine deletion phase to modify the machine deletion process. The following examples demonstrate possible ways that an Operator can use this functionality.
Example use cases for preDrain lifecycle hooks
- Proactively replacing machines
-
An Operator can use a
preDrainlifecycle hook to ensure that a replacement machine is successfully created and joined to the cluster before removing the instance of a deleted machine. This can mitigate the impact of disruptions during machine replacement or of replacement instances that do not initialize promptly. - Implementing custom draining logic
An Operator can use a
preDrainlifecycle hook to replace the machine controller draining logic with a different draining controller. By replacing the draining logic, the Operator would have more flexibility and control over the lifecycle of the workloads on each node.For example, the machine controller drain libraries do not support ordering, but a custom drain provider could provide this functionality. By using a custom drain provider, an Operator could prioritize moving mission-critical applications before draining the node to ensure that service interruptions are minimized in cases where cluster capacity is limited.
Example use cases for preTerminate lifecycle hooks
- Verifying storage detachment
-
An Operator can use a
preTerminatelifecycle hook to ensure that storage that is attached to a machine is detached before the machine is removed from the infrastructure provider. - Improving log reliability
After a node is drained, the log exporter daemon requires some time to synchronize logs to the centralized logging system.
A logging Operator can use a
preTerminatelifecycle hook to add a delay between when the node drains and when the machine is removed from the infrastructure provider. This delay would provide time for the Operator to ensure that the main workloads are removed and no longer adding to the log backlog. When no new data is being added to the log backlog, the log exporter can catch up on the synchronization process, thus ensuring that all application logs are captured.
6.2.5. Quorum protection with machine lifecycle hooks
For OpenShift Container Platform clusters that use the Machine API Operator, the etcd Operator uses lifecycle hooks for the machine deletion phase to implement a quorum protection mechanism.
By using a preDrain lifecycle hook, the etcd Operator can control when the pods on a control plane machine are drained and removed. To protect etcd quorum, the etcd Operator prevents the removal of an etcd member until it migrates that member onto a new node within the cluster.
This mechanism allows the etcd Operator precise control over the members of the etcd quorum and allows the Machine API Operator to safely create and remove control plane machines without specific operational knowledge of the etcd cluster.
6.2.5.1. Control plane deletion with quorum protection processing order
When a control plane machine is replaced on a cluster that uses a control plane machine set, the cluster temporarily has four control plane machines. When the fourth control plane node joins the cluster, the etcd Operator starts a new etcd member on the replacement node. When the etcd Operator observes that the old control plane machine is marked for deletion, it stops the etcd member on the old node and promotes the replacement etcd member to join the quorum of the cluster.
The control plane machine Deleting phase proceeds in the following order:
- A control plane machine is slated for deletion.
-
The control plane machine enters the
Deletingphase. To satisfy the
preDrainlifecycle hook, the etcd Operator takes the following actions:-
The etcd Operator waits until a fourth control plane machine is added to the cluster as an etcd member. This new etcd member has a state of
Runningbut notreadyuntil it receives the full database update from the etcd leader. - When the new etcd member receives the full database update, the etcd Operator promotes the new etcd member to a voting member and removes the old etcd member from the cluster.
After this transition is complete, it is safe for the old etcd pod and its data to be removed, so the
preDrainlifecycle hook is removed.-
The etcd Operator waits until a fourth control plane machine is added to the cluster as an etcd member. This new etcd member has a state of
-
The control plane machine status condition
Drainableis set toTrue. The machine controller attempts to drain the node that is backed by the control plane machine.
-
If draining fails,
Drainedis set toFalseand the machine controller attempts to drain the node again. -
If draining succeeds,
Drainedis set toTrue.
-
If draining fails,
-
The control plane machine status condition
Drainedis set toTrue. -
If no other Operators have added a
preTerminatelifecycle hook, the control plane machine status conditionTerminableis set toTrue. - The machine controller removes the instance from the infrastructure provider.
-
The machine controller deletes the
Nodeobject.
YAML snippet demonstrating the etcd quorum protection preDrain lifecycle hook
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
...
spec:
lifecycleHooks:
preDrain:
- name: EtcdQuorumOperator 1
owner: clusteroperator/etcd 2
...
