Chapter 2. Two-node with Fencing

Procedure

1. Configure the load balancer to forward traffic for the following ports:

   • 6443: Kubernetes API server

   • 80 and 443: Application ingress

     You must forward traffic to both control plane nodes.

2. Configure health checks on the load balancer. You must monitor the backend endpoints so that the load balancer only sends traffic to nodes that respond.

3. Configure the load balancer to forward traffic to both control plane nodes. The following example shows how to configure two control plane nodes:

   frontend api_frontend
       bind *:6443
       mode tcp
       default_backend api_backend
   
   backend api_backend
       mode tcp
       balance roundrobin
       server cp0 <cp0_ip>:6443 check
       server cp1 <cp1_ip>:6443 check
   
   frontend ingress_frontend
       bind *:80
       bind *:443
       mode tcp
       default_backend ingress_backend
   
   backend ingress_backend
       mode tcp
       balance roundrobin
       server cp0 <cp0_ip>:80 check
       server cp1 <cp1_ip>:80 check
       server cp0 <cp0_ip>:443 check
       server cp1 <cp1_ip>:443 check

   Copy to Clipboard Toggle word wrap

4. Verify the load balancer configuration:

   1. From an external client, run the following command:

      $ curl -k https://api.<cluster_name>.<base_domain>:6443/version

      Copy to Clipboard Toggle word wrap

   2. From an external client, access an application route by running the following command:

      $ curl https://<app>.<cluster_name>.<base_domain>

      Copy to Clipboard Toggle word wrap

1. Update fencing secrets: Refresh the Baseboard Management Console (BMC) credentials if they are incorrect or outdated.
2. Recover from a single-node failure: Restore functionality when only one control plane node is down.
3. Recover from a complete node failure: Restore functionality when both control plane nodes are down.
4. Replace a control plane node that cannot be recovered: Replace the node to restore cluster functionality.

Procedure

1. Update the fencing secrets:

   1. If the Cluster API is unavilable, update fencing secret by running the following command on one of the cluster nodes:

      $ sudo pcs stonith update <node_name>_redfish username=<user_name> password=<password>

      Copy to Clipboard Toggle word wrap

      After the Cluster API recovers, or the Cluster API is already available, update fencing secret in the cluster to ensure it stays in sync, as described in the following step.

   2. Edit the username and password for the existing fencing secret for the control plane node by running the following commads:

      $ oc project openshift-etcd

      Copy to Clipboard Toggle word wrap

      $ oc edit secret <node_name>-fencing

      Copy to Clipboard Toggle word wrap

      If the cluster recovers after updating the fencing secrets, no further action is required. If the issue persists, proceed to the next step.

2. Recover from a single-node failure:

   1. Gather initial diagnostics by running the following command:

      $ sudo pcs status --full

      Copy to Clipboard Toggle word wrap

      This command provides a detailed view of the current cluster and resource states. You can use the output to identify issues with fencing or etcd startup.

   2. Run the following additional diagnostic commands, if necessary:

      Reset the resources on your cluster and instruct Pacemaker to attempt to start them fresh by running the following command:

      $ sudo pcs resource cleanup

      Copy to Clipboard Toggle word wrap

      Review all Pacemaker activity on the node by running the following command:

      $ sudo journalctl -u pacemaker

      Copy to Clipboard Toggle word wrap

      Diagnose etcd resource startup issues by running the following command:

      $ sudo journalctl -u pacemaker | grep podman-etcd

      Copy to Clipboard Toggle word wrap

   3. View the fencing configuration for the node by running the following command:

      $ sudo pcs stonith config <node_name>_redfish

      Copy to Clipboard Toggle word wrap

      If fencing is required but is not functioning, ensure that the Redfish fencing endpoint is accessible and verify that the credentials are correct.

   4. If etcd is not starting despite fencing being operational, restore etcd from a backup by running the following commands:

      $ sudo cp -r /var/lib/etcd-backup/* /var/lib/etcd/

      Copy to Clipboard Toggle word wrap

      $ sudo chown -R etcd:etcd /var/lib/etcd

      Copy to Clipboard Toggle word wrap

      If the recovery is successful, no further action is required. If the issue persists, proceed to the next step.

3. Recover from a complete node failure:

   1. Power on both control plane nodes.

      Pacemaker starts automatically and begins the recovery operation when it detects both nodes are online. If the recovery does not start as expected, use the diagnostic commands described in the previous step to investigate the issue.

   2. Reset the resources on your cluster and instruct Pacemaker to attempt to start them fresh by running the following command:

      $ sudo pcs resource cleanup

      Copy to Clipboard Toggle word wrap

   3. Check resource start order by running the following command:

      $ sudo pcs status --full

      Copy to Clipboard Toggle word wrap

   4. Inspect the pacemaker service journal if kubelet fails by running the following commands:

      $ sudo journalctl -u pacemaker

      Copy to Clipboard Toggle word wrap

      $ sudo journalctl -u kubelet

      Copy to Clipboard Toggle word wrap

   5. Handle out-of-sync etcd.

      If one node has a more up-to-date etcd, Pacemaker attempts to fence the lagging node and start it as a learner. If this process stalls, verify the Redfish fencing endpoint and credentials by running the following command:

      $ sudo pcs stonith config

      Copy to Clipboard Toggle word wrap

      If the recovery is successful, no further action is required. If the issue persists, perform manual recovery as described in the next step.

4. If you need to manually recover from an event when one of the nodes is not recoverable, follow the procedure in "Replacing control plane nodes in a two-node OpenShift cluster".

   When a cluster loses a single node, it enters the degraded mode. In this state, Pacemaker automatically unblocks quorum and allows the cluster to temporarily operate on the remaining node.

   If both nodes fail, you must restart both nodes to reestablish quorum so that Pacemaker can resume normal cluster operations.

   If only one of the two nodes can be restarted, follow the node replacement procedure to manually reestablish quorum on the surviving node.

   If manual recovery is still required and it fails, collect a must-gather and SOS report, and file a bug.

Procedure

1. Check the quorum state by running the following command:

   $ sudo pcs quorum status

   Copy to Clipboard Toggle word wrap

   Example output

   Quorum information
   ------------------
   Date:             Fri Oct  3 14:15:31 2025
   Quorum provider:  corosync_votequorum
   Nodes:            2
   Node ID:          1
   Ring ID:          1.16
   Quorate:          Yes
   
   Votequorum information
   ----------------------
   Expected votes:   2
   Highest expected: 2
   Total votes:      2
   Quorum:           1
   Flags:            2Node Quorate WaitForAll
   
   Membership information
   ----------------------
       Nodeid      Votes    Qdevice Name
            1          1         NR master-0 (local)
            2          1         NR master-1

   Copy to Clipboard Toggle word wrap

   1. If quorum is lost and one control plane node is still running, restore quorum manually on the survivor node by running the following command:

      $ sudo pcs quorum unblock

      Copy to Clipboard Toggle word wrap

   2. If only one node failed, verify that etcd is running on the survivor node by running the following command:

      $ sudo pcs resource status etcd

      Copy to Clipboard Toggle word wrap

   3. If etcd is not running, restart etcd by running the following command:

      $ sudo pcs resource cleanup etcd

      Copy to Clipboard Toggle word wrap

      If etcd still does not start, force it manually on the survivor node, skipping fencing:

      Important

      Before running this commands, ensure that the node being replaced is inaccessible. Otherwise, you risk etcd corruption.

      $ sudo pcs resource debug-stop etcd

      Copy to Clipboard Toggle word wrap

      $ sudo OCF_RESKEY_CRM_meta_notify_start_resource='etcd' pcs resource debug-start etcd

      Copy to Clipboard Toggle word wrap

      After recovery, etcd must be running successfully on the survivor node.

2. Delete etcd secrets for the failed node by running the following commands:

   $ oc project openshift-etcd

   Copy to Clipboard Toggle word wrap

   $ oc delete secret etcd-peer-<node_name>

   Copy to Clipboard Toggle word wrap

   $ oc delete secret etcd-serving-<node_name>

   Copy to Clipboard Toggle word wrap

   $ oc delete secret etcd-serving-metrics-<node_name>

   Copy to Clipboard Toggle word wrap
   Note

   To replace the failed node, you must delete its etcd secrets first. When etcd is running, it might take some time for the API server to respond to these commands.

3. Delete resources for the failed node:

   1. If you have the BareMetalHost (BMH) objects, list them to identify the host you are replacing by running the following command:

      $ oc get bmh -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

   2. Delete the BMH object for the failed node by running the following command:

      $ oc delete bmh/<bmh_name> -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

   3. List the Machine objects to identify the object that maps to the node that you are replacing by running the following command:

      $ oc get machines.machine.openshift.io -n openshift-machine-api

      Copy to Clipboard Toggle word wrap

   4. Get the label with the machine hash value from the Machine object by running the following command:

      $ oc get machines.machine.openshift.io/<machine_name> -n openshift-machine-api \
        -o jsonpath='Machine hash label: {.metadata.labels.machine\.openshift\.io/cluster-api-cluster}{"\n"}'

      Copy to Clipboard Toggle word wrap

      Replace <machine_name> with the name of a Machine object in your cluster. For example, ostest-bfs7w-ctrlplane-0.

      You need this label to provision a new Machine object.

   5. Delete the Machine object for the failed node by running the following command:

      $ oc delete machines.machine.openshift.io/<machine_name>-<failed nodename> -n openshift-machine-api

      Copy to Clipboard Toggle word wrap
      Note

      The node object is deleted automatically after deleting the Machine object.

4. Recreate the failed host by using the same name and IP address:

   Important

   You must perform this step only if you are using installer-provisioned infrastructure or the Machine API to create the original node. For information about replacing a failed bare-metal control plane node, see "Replacing an unhealthy etcd member on bare metal".

   1. Remove the BMH and Machine objects. The machine controller automatically deletes the node object.

   2. Provision a new machine by using the following sample configuration:

      Example Machine object configuration

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          metal3.io/BareMetalHost: openshift-machine-api/{bmh_name}
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: {machine_hash_label}
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: {machine_name}
        namespace: openshift-machine-api
      spec:
        authoritativeAPI: MachineAPI
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
              name: master-user-data-managed

      Copy to Clipboard Toggle word wrap

      • metadata.annotations.metal3.io/BareMetalHost: Replace {bmh_name} with the name of the BMH object that is associated with the host that you are replacing.
      • labels.machine.openshift.io/cluster-api-cluster: Replace {machine_hash_label} with the label that you fetched from the machine you deleted.
      • metadata.name: Replace {machine_name} with the name of the machine you deleted.

   3. Create the new BMH object and the secret to store the BMC credentials by running the following command:

      cat <<EOF | oc apply -f -
      apiVersion: v1
      kind: Secret
      metadata:
        name: <secret_name>
        namespace: openshift-machine-api
      data:
        password: <password>
        username: <username>
      type: Opaque
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: {bmh_name}
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: disabled
        bmc:
          address: <redfish_url>/{uuid}
          credentialsName: <name>
          disableCertificateVerification: true
        bootMACAddress: {boot_mac_address}
        bootMode: UEFI
        externallyProvisioned: false
        online: true
        rootDeviceHints:
          deviceName: /dev/disk/by-id/scsi-<serial_number>
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api
      EOF

      Copy to Clipboard Toggle word wrap
      • metadata.name: Specify the name of the secret.
      • metadata.name: Replace {bmh_name} with the name of the BMH object that you deleted.
      • bmc.address: Replace {uuid} with the UUID of the node that you created.
      • bmc.credentialsName: Replace name with the name of the secret that you created.
      • bootMACAddress: Specify the MAC address of the provisioning network interface. This is the MAC address the node uses to identify itself when communicating with Ironic during provisioning.

5. Verify that the new node has reached the Provisioned state by running the following command:

   $ oc get bmh -o wide

   Copy to Clipboard Toggle word wrap

   The value of the STATUS column in the output of this command must be Provisioned.

   Note

   The provisioning process can take 10 to 20 minutes to complete.

6. Verify that both control plane nodes are in the Ready state by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   The value of the STATUS column in the output of this command must be Ready for both nodes.

7. Apply the detached annotation to the BMH object to prevent the Machine API from managing it by running the following command:

   $ oc annotate bmh <bmh_name> -n openshift-machine-api baremetalhost.metal3.io/detached='' --overwrite

   Copy to Clipboard Toggle word wrap

8. Rejoin the replacement node to the pacemaker cluster by running the following command:

   Note

   Run the following command on the survivor control plane node, not the node being replaced.

   $ sudo pcs cluster node remove <node_name>

   Copy to Clipboard Toggle word wrap

   $ sudo pcs cluster node add <node_name> addr=<node_ip> --start --enable

   Copy to Clipboard Toggle word wrap

9. Delete stale jobs for the failed node by running the following command:

   $ oc project openshift-etcd

   Copy to Clipboard Toggle word wrap

   $ oc delete job tnf-auth-job-<node_name>

   Copy to Clipboard Toggle word wrap

   $ oc delete job tnf-after-setup-job-<node_name>

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the overall node status by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   This command verifies that both control plane nodes are in the Ready state, indicating that they can receive workloads for scheduling.

2. Verify the status of the cluster-etcd-operator by running the following command:

   $ oc describe co/etcd

   Copy to Clipboard Toggle word wrap

   The cluster-etcd-operator manages and reports on the health of your etcd setup. Reviewing its status helps you identify any ongoing issues or degraded conditions.

3. Review the etcd member list by running the following command:

   $ oc rsh -n openshift-etcd <etcd_pod> etcdctl member list -w table

   Copy to Clipboard Toggle word wrap

   This command shows the current etcd members and their roles. Look for any nodes marked as learner, which indicates that they are in the process of becoming voting members.

4. Review the Pacemaker resource status by running the following command on either control plane node:

   $ sudo pcs status --full

   Copy to Clipboard Toggle word wrap

   This command provides a detailed overview of all resources managed by Pacemaker. You must ensure that the following conditions are met:

   • Both nodes are online.
   • The kubelet and etcd resources are running.
   • Fencing is correctly configured for both nodes.