Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 3. Using Fence Agents Remediation

You can use the Fence Agents Remediation Operator to automatically remediate unhealthy nodes, similar to the Self Node Remediation Operator. Using a management interface or traditional API, this Operator runs a fence-agent to remediate a node from an unhealthy state by power-cycling the node.

3.1. About the Fence Agents Remediation Operator
Copy link

The Fence Agents Remediation (FAR) Operator uses external tools to fence unhealthy nodes. These tools are a set of fence agents, where each fence agent can be used for different environments to fence a node, and using a traditional Application Programming Interface (API) call that reboots a node. By doing so, FAR can minimize downtime for stateful applications, restores compute capacity if transient failures occur, and increases the availability of workloads.

FAR not only fences a node when it becomes unhealthy, it also tries to remediate the node from being unhealthy to healthy. It adds a taint to evict stateless pods, fences the node with a fence agent, and after a reboot, it completes the remediation with resource deletion to remove any remaining workloads (mostly stateful workloads). Adding the taint and deleting the workloads accelerates the workload rescheduling.

The Operator watches for new or deleted custom resources (CRs) called FenceAgentsRemediation which trigger a fence agent to remediate a node, based on the CR’s name. FAR uses the NodeHealthCheck controller to detect the health of a node in the cluster. When a node is identified as unhealthy, the NodeHealthCheck resource creates the FenceAgentsRemediation CR, based on the FenceAgentsRemediationTemplate CR, which then triggers the Fence Agents Remediation Operator.

FAR uses a fence agent to fence a Kubernetes node. Generally, fencing is the process of taking unresponsive/unhealthy computers into a safe state, and isolating the computer. Fence agent is a software code that uses a management interface to perform fencing, mostly power-based fencing which enables power-cycling, reset, or turning off the computer. An example fence agent is fence_ipmilan which is used for Intelligent Platform Management Interface (IPMI) environments. 

apiVersion: fence-agents-remediation.medik8s.io/v1alpha1
kind: FenceAgentsRemediation
metadata:
  name: node-name
1 

  namespace: openshift-operators
spec:
1
The node-name should match the name of the unhealthy cluster node.

The Operator includes a set of fence agents, that are also available in the Red Hat High Availability Add-On, which use a management interface, such as IPMI or an API, to provision/reboot a node for bare metal servers, virtual machines, and cloud platforms.

You can use the Red Hat OpenShift web console to install the Fence Agents Remediation Operator.

Prerequisites

  • Log in as a user with cluster-admin privileges.

Procedure

  1. In the Red Hat OpenShift web console, navigate to Operators OperatorHub.
  2. Select the Fence Agents Remediation Operator, or FAR, from the list of available Operators, and then click Install.
  3. Keep the default selection of Installation mode and namespace to ensure that the Operator is installed to the openshift-operators namespace.
  4. Click Install.

Verification

To confirm that the installation is successful:

  1. Navigate to the Operators Installed Operators page.
  2. Check that the Operator is installed in the openshift-operators namespace and its status is Succeeded.

If the Operator is not installed successfully:

  1. Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures.
  2. Navigate to the Workloads Pods page and check the log of the fence-agents-remediation-controller-manager pod for any reported issues.

You can use the OpenShift CLI (oc) to install the Fence Agents Remediation Operator.

You can install the Fence Agents Remediation Operator in your own namespace or in the openshift-operators namespace.

To install the Operator in your own namespace, follow the steps in the procedure.

To install the Operator in the openshift-operators namespace, skip to step 3 of the procedure because the steps to create a new Namespace custom resource (CR) and an OperatorGroup CR are not required.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a Namespace custom resource (CR) for the Fence Agents Remediation Operator:

    1. Define the Namespace CR and save the YAML file, for example, fence-agents-remediation-namespace.yaml: 

      apiVersion: v1
kind: Namespace
metadata:
  name: fence-agents-remediation-namespace

    2. To create the Namespace CR, run the following command: 

      $ oc create -f fence-agents-remediation-namespace.yaml

  2. Create an OperatorGroup CR:

    1. Define the OperatorGroup CR and save the YAML file, for example, fence-agents-remediation-operator-group.yaml: 

      apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: fence-agents-remediation-operator-group
  namespace: fence-agents-remediation-namespace

    2. To create the OperatorGroup CR, run the following command: 

      $ oc create -f fence-agents-remediation-operator-group.yaml

  3. Create a Subscription CR:

    1. Define the Subscription CR and save the YAML file, for example, fence-agents-remediation-subscription.yaml: 

      apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
    name: fence-agents-remediation-subscription
    namespace: fence-agents-remediation-namespace
      1 
      
spec:
    channel: stable
    name: fence-agents-remediation
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    package: fence-agents-remediation
      1
      Specify the Namespace where you want to install the Fence Agents Remediation Operator, for example, the fence-agents-remediation-namespace outlined earlier in this procedure. You can install the Subscription CR for the Fence Agents Remediation Operator in the openshift-operators namespace where there is already a matching OperatorGroup CR.

    2. To create the Subscription CR, run the following command: 

      $ oc create -f fence-agents-remediation-subscription.yaml

Verification

  1. Verify that the installation succeeded by inspecting the CSV resource: 

    $ oc get csv -n fence-agents-remediation-namespace

    Example output

    NAME                               DISPLAY                          VERSION   REPLACES   PHASE
fence-agents-remediation.v.0.2.0      Fence Agents Remediation Operator   v.0.2.0              Succeeded

  2. Verify that the Fence Agents Remediation Operator is up and running: 

    $ oc get deploy -n fence-agents-remediation-namespace

    Example output

    NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
fence-agents-remediation-controller-manager    1/1     1            1           28h

You can use the Fence Agents Remediation Operator to create the FenceAgentsRemediationTemplate Custom Resource (CR), which is used by the Node Health Check Operator (NHC). This CR defines the fence agent to be used in the cluster with all the required parameters for remediating the nodes. There may be many FenceAgentsRemediationTemplate CRs, at most one for each fence agent, and when NHC is being used it can choose the FenceAgentsRemediationTemplate as the remediationTemplate to be used for power-cycling the node.

Note

In the current release, there may be many FenceAgentsRemediationTemplate CRs, but at most one for each fence agent. This is as a known limitation that will be addressed in a future release.

The FenceAgentsRemediationTemplate CR resembles the following YAML file: 

apiVersion: fence-agents-remediation.medik8s.io/v1alpha1
kind: FenceAgentsRemediationTemplate
metadata:
  name: fence-agents-remediation-template-fence-ipmilan
  namespace: openshift-operators
spec:
  template:
    spec:
      agent: fence_ipmilan
1 

        nodeparameters:
2 

          --ipport:
            master-0-0: '6230'
            master-0-1: '6231'
            master-0-2: '6232'
            worker-0-0: '6233'
            worker-0-1: '6234'
            worker-0-2: '6235'
        sharedparameters:
3 

          '--action': reboot
          '--ip': 192.168.123.1
          '--lanplus': ''
          '--password': password
          '--username': admin
1
Displays the name of fence agent to be executed, for example, fence_ipmilan.
2
Displays the specific cluster nodes information for executing the fence agent, for example, ipport.
3
Displays the common cluster nodes information for executing the fence agent, for example, username.

3.5.1. General troubleshooting
Copy link

Issue
You want to troubleshoot issues with the Fence Agents Remediation Operator.
Resolution

Check the Operator logs. 

$ oc logs <fence-agents-remediation-controller-manager-name> -c manager -n <namespace-name>

3.5.2. Unsuccessful remediation
Copy link

Issue
An unhealthy node was not remediated.
Resolution

Verify that the FenceAgentsRemediation CR was created by running the following command: 

$ oc get far -A

If the NodeHealthCheck controller did not create the FenceAgentsRemediation CR when the node turned unhealthy, check the logs of the NodeHealthCheck controller. Additionally, ensure that the NodeHealthCheck CR includes the required specification to use the remediation template.

If the FenceAgentsRemediation CR was created, ensure that its name matches the unhealthy node object.

Issue
The Fence Agents Remediation Operator resources, such as the remediation CR and the remediation template CR, exist after uninstalling the Operator.
Resolution

To remove the Fence Agents Remediation Operator resources, you can delete the resources by selecting the "Delete all operand instances for this operator" checkbox before uninstalling. This checkbox feature is only available in Red Hat OpenShift since version 4.13. For all versions of Red Hat OpenShift, you can delete the resources by running the following relevant command for each resource type: 

$ oc delete far <fence-agents-remediation> -n <namespace>
 
$ oc delete fartemplate <fence-agents-remediation-template> -n <namespace>

The remediation CR far must be created and deleted by the same entity, for example, NHC. If the remediation CR far is still present, it is deleted, together with the FAR operator.

The remediation template CR fartemplate only exists if you use FAR with NHC. When the FAR operator is deleted using the web console, the remediation template CR fartemplate is also deleted.

To collect debugging information about the Fence Agents Remediation Operator, use the must-gather tool. For information about the must-gather image for the Fence Agents Remediation Operator, see Gathering data about specific features.

3.7. Additional resources
Copy link

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top