Configure a node health strategy

A node might be functioning well enough to maintain its cluster membership and yet be unhealthy in some respect that makes it an undesirable location for resources. For example, a disk drive might be reporting SMART errors, or the CPU might be highly loaded.

About this task

You can use a node health strategy in Pacemaker to automatically move resources off unhealthy nodes.

You can monitor a node’s health with the following health node resource agents, which set node attributes based on CPU and disk status:

  • ocf:pacemaker:HealthCPU, which monitors CPU idling
  • ocf:pacemaker:HealthIOWait, which monitors the CPU I/O wait
  • ocf:pacemaker:HealthSMART, which monitors SMART status of a disk drive
  • ocf:pacemaker:SysInfo, which sets a variety of node attributes with local system information and also functions as a health agent monitoring disk space usage

Additionally, any resource agent might provide node attributes that can be used to define a health node strategy.

The following procedure configures a health node strategy for a cluster that will move resources off of any node whose CPU I/O wait goes above 15%.

Procedure

  1. Set the health-node-strategy cluster property to define how Pacemaker responds to changes in node health. 
    # pcs property set node-health-strategy=migrate-on-red
  2. Create a cloned cluster resource that uses a health node resource agent, setting the allow-unhealthy-nodes resource meta option to define whether the cluster will detect if the node’s health recovers and move resources back to the node. Configure this resource with a recurring monitor action, to continually check the health of all nodes.

    This example creates a HealthIOWait resource agent to monitor the CPU I/O wait, setting a red limit for moving resources off a node to 15%. This command sets the allow-unhealthy-nodes resource meta option to true and configures a recurring monitor interval of 10 seconds.

    # pcs resource create io-monitor ocf:pacemaker:HealthIOWait red_limit=15 op monitor interval=10s meta allow-unhealthy-nodes=true clone