Chapter 6. Remediating nodes with Node Health Checks

Procedure

1. In the Red Hat OpenShift web console, navigate to Operators OperatorHub.
2. Select the Node Health Check Operator, then click Install.
3. Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-workload-availability namespace.
4. Ensure that the Console plug-in is set to Enable.
5. Click Install.

1. Navigate to the Operators Installed Operators page.
2. Check that the Operator is installed in the openshift-workload-availability namespace and that its status is Succeeded.

1. Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures.
2. Navigate to the Workloads Pods page and check the logs in any pods in the openshift-workload-availability project that are reporting issues.

Procedure

1. Create a Namespace custom resource (CR) for the Node Health Check Operator:

   1. Define the Namespace CR and save the YAML file, for example, node-health-check-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-workload-availability

      Copy to Clipboard Toggle word wrap

   2. To create the Namespace CR, run the following command:

      $ oc create -f node-health-check-namespace.yaml

      Copy to Clipboard Toggle word wrap

2. Create an OperatorGroup CR:

   1. Define the OperatorGroup CR and save the YAML file, for example, workload-availability-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: workload-availability-operator-group
        namespace: openshift-workload-availability

      Copy to Clipboard Toggle word wrap

   2. To create the OperatorGroup CR, run the following command:

      $ oc create -f workload-availability-operator-group.yaml

      Copy to Clipboard Toggle word wrap

3. Create a Subscription CR:

   1. Define the Subscription CR and save the YAML file, for example, node-health-check-subscription.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: node-health-check-operator
          namespace: openshift-workload-availability 

      1

      
      spec:
          channel: stable 

      2

      
          installPlanApproval: Manual 

      3

      
          name: node-healthcheck-operator
          source: redhat-operators
          sourceNamespace: openshift-marketplace
          package: node-healthcheck-operator

      Copy to Clipboard Toggle word wrap

      1

      Specify the Namespace where you want to install the Node Health Check Operator. To install the Node Health Check Operator in the openshift-workload-availability namespace, specify openshift-workload-availability in the Subscription CR.

      2

      Specify the channel name for your subscription. To upgrade to the latest version of the Node Health Check Operator, you must manually change the channel name for your subscription from candidate to stable.

      3

      Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2. To create the Subscription CR, run the following command:

      $ oc create -f node-health-check-subscription.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. Verify that the installation succeeded by inspecting the CSV resource:

   $ oc get csv -n openshift-workload-availability

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                              DISPLAY                     VERSION  REPLACES PHASE
   node-healthcheck-operator.v0.7.0  Node Health Check Operator  0.7.0   node-healthcheck-operator.v0.6.1           Succeeded

   Copy to Clipboard Toggle word wrap

2. Verify that the Node Health Check Operator is up and running:

   $ oc get deployment -n openshift-workload-availability

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
   node-healthcheck-controller-manager            2/2     2            2           10d

   Copy to Clipboard Toggle word wrap

Procedure

1. From the Administrator perspective of the Red Hat OpenShift web console, click Compute NodeHealthChecks CreateNodeHealthCheck.
2. Specify whether to configure the node health check using the Form view or the YAML view.
3. Enter a Name for the node health check. The name must consist of lower case, alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character.
4. Specify the Remediator type, as Self node remediation or Other. You must install the Self Node Remediation Operator manually. Selecting Other requires an API version, Kind, Name, and Namespace to be entered, which then points to the remediation template resource of a remediator.

5. Make a Nodes selection by specifying the labels of the nodes you want to remediate. The selection matches labels that you want to check. If more than one label is specified, the nodes must contain each label. The default value is empty, which selects both worker and control-plane nodes.

   Note

   When creating a node health check with the Self Node Remediation Operator, you must select either node-role.kubernetes.io/worker or node-role.kubernetes.io/control-plane as the value.

6. Specify the minimum number of healthy nodes, using either a percentage or a number, required for a NodeHealthCheck to remediate nodes in the targeted pool. If the number of healthy nodes equals to or exceeds the limit set by Min healthy, remediation occurs. The default value is 51%.
7. Specify a list of Unhealthy conditions that if a node meets determines whether the node is considered unhealthy, and requires remediation. You can specify the Type, Status and Duration. You can also create your own custom type.
8. Click Create to create the node health check.