Chapter 3. Using Fence Agents Remediation

The Fence Agents Remediation (FAR) Operator uses external tools to fence unhealthy nodes. These tools are a set of fence agents, where each fence agent can be used for different environments to fence a node, and using a traditional Application Programming Interface (API) call that fences a node, usually by rebooting the node. By doing so, FAR can minimize downtime for stateful applications, restore compute capacity if transient failures occur, and increase the availability of workloads.

FAR not only fences a node when it becomes unhealthy, it also tries to remediate the node from being unhealthy to healthy. It adds a taint to stop the scheduling of new pods, fences the node with a fence agent, and when fencing is finished, it completes the remediation with resource deletion to remove any remaining workloads (mostly stateful workloads). Adding the taint and deleting the workloads accelerates the workload rescheduling.

FAR supports two fencing actions 1) the reboot action reboots the specific node, so it rejoins the cluster when fencing is complete, and 2) the off action stops the node by shutting it down, and ensures fencing without recovering the node. When using the off action, the administrator needs to manually turn on the node to rejoin the cluster.

The Operator watches for new or deleted custom resources (CRs) called FenceAgentsRemediation which trigger a fence agent to remediate a node, based on the CR’s name. FAR uses the NodeHealthCheck controller to detect the health of a node in the cluster. When a node is identified as unhealthy, the NodeHealthCheck resource creates the FenceAgentsRemediation CR, based on the FenceAgentsRemediationTemplate CR, which then triggers the Fence Agents Remediation Operator.

FAR uses a fence agent to fence a Kubernetes node. Generally, fencing is the process of taking unresponsive/unhealthy computers into a safe state, and isolating the computer. A fence agent is a software code that uses a management interface to perform fencing, mostly power-based fencing which enables power-cycling, reset, or turning off the computer. An example fence agent is fence_ipmilan which is used for Intelligent Platform Management Interface (IPMI) environments.

apiVersion: fence-agents-remediation.medik8s.io/v1alpha1
kind: FenceAgentsRemediation
metadata:
  name: node-name 

  namespace: openshift-workload-availability
spec:
  remediationStrategy: <remediation_strategy> 

The Operator includes a set of fence agents, that are also available in the Red Hat High Availability Add-On, which use a management interface, such as IPMI or an API, to shutdown/provision/reboot a node for bare metal servers, virtual machines, and cloud platforms.

You can use the Red Hat OpenShift web console to install the Fence Agents Remediation Operator.

If the Operator is not installed successfully:

You can use the OpenShift CLI (oc) to install the Fence Agents Remediation Operator.

You can install the Fence Agents Remediation Operator in your own namespace or in the openshift-workload-availability namespace.

You can use the Fence Agents Remediation Operator to create the FenceAgentsRemediationTemplate Custom Resource (CR), which is used by the Node Health Check Operator (NHC). This CR defines the fence agent to be used in the cluster with all the required parameters for remediating the nodes. There may be many FenceAgentsRemediationTemplate CRs, at most one for each fence agent, and when NHC is being used it can choose the FenceAgentsRemediationTemplate as the remediationTemplate to be used for power-cycling the node.

The FenceAgentsRemediationTemplate CR resembles the following YAML file:

apiVersion: fence-agents-remediation.medik8s.io/v1alpha1
kind: FenceAgentsRemediationTemplate
metadata:
  name: fence-agents-remediation-template-fence-ipmilan
  namespace: openshift-workload-availability
spec:
  template:
    spec:
      agent: fence_ipmilan 

      nodeparameters: 

        --ipport:
          master-0-0: '6230'
          master-0-1: '6231'
          master-0-2: '6232'
          worker-0-0: '6233'
          worker-0-1: '6234'
          worker-0-2: '6235'
      nodeSecretNames: 

        master-0-0: fence-agents-credentials-master-0-0
        worker-0-0: fence-agents-credentials-worker-0-0
        worker-0-1: fence-agents-credentials-worker-0-1
      sharedparameters: 

        '--action': reboot 

        '--ip': 192.168.123.1
        '--lanplus': ''
        '--password': password
        '--username': admin
      retrycount: '5' 

      retryinterval: '5s' 

      sharedSecretName: 'fence-agents-credentials-shared' 

      timeout: '60s' 

apiVersion: fence-agents-remediation.medik8s.io/v1alpha1
kind: FenceAgentsRemediationTemplate
metadata:
  name: fence-agents-remediation-<remediation_object>-deletion-template 

  namespace: openshift-workload-availability
spec:
  template:
    spec:
      remediationStrategy: <remediation_strategy>  

This section describes the agents currently supported by the Fence Agents Remediation Operator.

Most of the supported agents can be grouped by the node’s hardware proprietary and usage, as follows: