Chapter 8. Deploying machine health checks
You can configure and deploy a machine health check to automatically repair damaged machines in a machine pool.
This process is not applicable to clusters where you manually provisioned the machines yourself. You can use the advanced machine management and scaling capabilities only in clusters where the machine API is operational.
8.1. About MachineHealthChecks
MachineHealthChecks automatically repairs unhealthy Machines in a particular MachinePool.
To monitor machine health, you create a resource to define the configuration for a controller. You set a condition to check for, such as staying in the NotReady status for 15 minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.
You cannot apply a MachineHealthCheck to a machine with the master role.
The controller that observes a MachineHealthCheck resource checks for the status that you defined. If a machine fails the health check, it is automatically deleted and a new one is created to take its place. When a machine is deleted, you see a machine deleted event. To limit disruptive impact of the machine deletion, the controller drains and deletes only one node at a time. If there are more unhealthy machines than the maxUnhealthy threshold allows for in the targeted pool of machines, remediation stops so that manual intervention can take place.
To stop the check, you remove the resource.
8.2. Sample MachineHealthCheck resource
The MachineHealthCheck resource resembles the following YAML file:
MachineHealthCheck
apiVersion: machine.openshift.io/v1beta1 kind: MachineHealthCheck metadata: name: example 1 namespace: openshift-machine-api spec: selector: matchLabels: machine.openshift.io/cluster-api-machine-role: <role> 2 machine.openshift.io/cluster-api-machine-type: <role> 3 machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone> 4 unhealthyConditions: - type: "Ready" timeout: "300s" 5 status: "False" - type: "Ready" timeout: "300s" 6 status: "Unknown" maxUnhealthy: "40%" 7
- 1
- Specify the name of the MachineHealthCheck to deploy.
- 2 3
- Specify a label for the machine pool that you want to check.
- 4
- Specify the MachineSet to track in
<cluster_name>-<label>-<zone>format. For example,prod-node-us-east-1a. - 5 6
- Specify the timeout duration for a node condition. If a condition is met for the duration of the timeout, the Machine will be remediated. Long timeouts can result in long periods of downtime for the workload on the unhealthy Machine.
- 7
- Specify the amount of unhealthy machines allowed in the targeted pool of machines. This can be set as a percentage or an integer.
The matchLabels are examples only; you must map your machine groups based on your specific needs.
8.3. Creating a MachineHealthCheck resource
You can create a MachineHealthCheck resource for all MachinePools in your cluster except the master pool.
Prerequisites
-
Install the
occommand line interface.
Procedure
-
Create a
healthcheck.ymlfile that contains the definition of your MachineHealthCheck. Apply the
healthcheck.ymlfile to your cluster:$ oc apply -f healthcheck.yml
