Installing a Two Node OpenShift Cluster

Procedure

1. Power on the failed node and verify network connectivity on all three network planes before the node can fully rejoin the cluster.

   • BMC or management network: Ensure that the peer node must reach the BMC address of the failed node. Without BMC connectivity, fencing cannot protect the cluster if a subsequent failure occurs.
   • Cluster network: Ensure that there is bilateral connectivity between nodes for Corosync membership (ports 5405 to 5407), Pacemaker management (port 2224), and etcd replication (ports 2379, 2380).
   • Kubernetes API: Ensure that the recovering node can reach the API server (port 6443).
2. Wait for Pacemaker to re-establish communication. Corosync detects the returning node and Pacemaker marks it as ONLINE. Pacemaker then starts managed resources, including kubelet and etcd, on the returning node.

3. Wait for etcd to re-join the cluster. The returning member first rejoins as a learner (non-voting), receives a data snapshot, and replicates the log from the surviving member. After it has caught up, it is promoted to a voting member, and the cluster returns to two-member quorum.

   Note

   This process typically takes about 15 to 25 minutes.

4. Verify that the node transitions to Ready status by running the following command:

   $ oc get nodes

Verification

1. Verify that cluster Operators have cleared their degraded conditions by running the following command:

   $ oc get co

2. After recovery, confirm the following conditions:

   • Both nodes show Ready status by running the following command:

     $ oc get nodes

     The output is similar to the following example:

     oc get nodes
     NAME                                                   STATUS   ROLES                         AGE    VERSION
     e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com   Ready    control-plane,master,worker   5d1h   v1.35.4
     e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com   Ready    control-plane,master,worker   5d1h   v1.35.4

   • Verify that the etcd Operator no longer reports Degraded=True by running the following command:

     $ oc get co etcd

     The output is similar to the following example:

     NAME   VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
     etcd   4.22.0-rc.3   True        False         False      5d1h

   • Verify that both fencing devices are operational.
   • Queued MachineConfig changes, upgrades, and certificate rotations can proceed.

Procedure

1. Check Pacemaker status from the surviving node by running the following command:

   $ oc debug node/<surviving-node> -- chroot /host pcs status

2. Check etcd membership by running the following command:

   $ oc debug node/<surviving-node> -- chroot /host podman exec etcd etcdctl member list -w table

3. Check node status by running the following command:

   $ oc get nodes

4. Check cluster Operators by running the following command:

   $ oc get co

Procedure

1. Verify that the failed node is powered off using the BMC console or physical inspection.

2. Confirm the fencing manually by running the following command:

   $ oc debug node/<surviving-node> -- chroot /host pcs stonith confirm <failed_node_name> --force

Procedure

1. Clean up the etcd resource by running the following command:

   $ oc debug node/<surviving_node> -- chroot /host bash -c '
     pcs resource cleanup etcd
   '

   The output is similar to the following example:

   Example

   sudo  pcs resource cleanup etcd
   +
   Cleaned up etcd:0 on e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   Cleaned up etcd:1 on e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   Waiting for 1 reply from the controller
   +
   ... got reply (done)

2. Verify Pacemaker status by running the following command:

   $ oc debug node/<surviving_node> -- chroot /host bash -c ' \
     pcs status
   '

   The output is similar to the following example:

   Cluster name: TNF
   +
   Cluster Summary:
   +
   Stack: corosync (Pacemaker is running)
   +
   Current DC: e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com (version 2.1.10-2.el9-5693eaeee) - partition with quorum
   +
   Last updated: Wed May 20 17:36:23 2026 on e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   Last change:  Wed May 20 17:36:22 2026 by root via root on e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   2 nodes configured
   +
   6 resource instances configured
   +
   Node List:
   +
   Online: [ e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com ]
   +
   Full List of Resources:
   +
   Clone Set: kubelet-clone [kubelet]:
   +
   Started: [ e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com ]
   +
   e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com_redfish	(stonith:fence_redfish):	 Started e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com_redfish	(stonith:fence_redfish):	 Started e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com
   +
   Clone Set: etcd-clone [etcd]:
   +
   Started: [ e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com ]
   +
   Failed Resource Actions:
   +
   e920t-01.cluster1.metal-platform.eng.rdu2.redhat.com_redfish 1m-interval monitor on e920t-02.cluster1.metal-platform.eng.rdu2.redhat.com could not be executed (Timed Out: Fence agent did not complete within 20s) at Sun May 17 12:50:32 2026 after 20.003s
   +
   Daemon Status:
   +
   corosync: active/enabled
   +
   pacemaker: active/enabled
   +
   pcsd: active/enabled

Procedure

1. Verify Corosync and cluster service status on the returning node by running the following command:

   $ oc debug node/<returning-node> -- chroot /host bash -c '\
     corosync-cfgtool -s\
     systemctl status corosync pacemaker pcsd
   '

2. Verify network connectivity between both nodes.

   • For a Cluster Network (Corosync), ping an adjacent node to check the peer node on the cluster network by running the following command:

     $ ping -c 3 <peer_node_ip>

   • For a Management or BMC Network (Fencing Path), run the following command:

     $ ping -c 3

     The output is similar to the following example:

     HTTP_CODE: 401
     TIME_TOTAL: 0.224628s
     TIME_CONNECT: 0.000322s

     Note

     The HTTP 401 code is expected because the curl command does not pass credentials (no --user Administrator:password). It confirms the Redfish API is listening and rejecting unauthenticated requests.

     TIME_CONNECT value is approximately 0.322ms which shows the fast connectivity.

   • For an Application Network (OpenShift/etcd), perform the following tasks:

     • Check API server health on the local node by running the following command:

       $ curl -sk https://localhost:6443/healthz

     • Check API server health on the peer node by running the following command:

       $ curl -sk https://<peer-node-ip>:6443/healthz

     • List etcd cluster members by running the following command:

       $ podman exec etcd etcdctl member list --write-out=table

     • Check etcd endpoint health across the cluster by running the following command:

       $ podman exec etcd etcdctl endpoint health --cluster

1. Update fencing secrets: Refresh the Baseboard Management Console (BMC) credentials if they are incorrect or outdated.
2. Recover from a single-node failure: Restore functionality when only one control plane node is down.
3. Recover from dual node power loss: Restore functionality when both control plane nodes are down and can be restarted.
4. Restore corosync quorum after dual node power loss: Restore corosync quorum when both control plane nodes lost power but only one node can be restarted.
5. Replace a control plane node that cannot be recovered: Replace the node to restore cluster functionality.

Procedure

1. Update the fencing secrets:

   1. If the Cluster API is unavailable, update the fencing secret by running the following command on one of the cluster nodes:

      $ sudo pcs stonith update <node_name>_redfish username=<user_name> password=<password>

      After the Cluster API recovers, or if the Cluster API is already available, update the fencing secret in the cluster to ensure it stays in sync, as described in the following step.

   2. Edit the username and password for the existing fencing secret for the control plane node by running the following commands:

      $ oc project openshift-etcd

      $ oc edit secret <node_name>-fencing
      $ oc edit secret fencing-credentials-<node_name>

      The secret contains the following data keys:

      Expand

      Table 2.6. Data keys

      Key	Description	Changes during credential rotation?
      username	BMC username	Yes
      password	BMC password	Yes
      address	Full Redfish URL (for example, redfish+https://192.168.1.10:443/redfish/v1/Systems/1)	Only if BMC address changed
      certificateVerification	Disabled or Enabled	Only if TLS settings changed

      Show more
      Note

      The oc edit secret command displays base64-encoded values. If you modify any of the values, the new values must also be base64-encoded.

      Alternatively, you can use the following command to create or update the secret with literal strings:

      $ oc create secret generic fencing-credentials-<node_name> \
        --from-literal=address='<redfish_address>' \
        --from-literal=username='<new_username>' \
        --from-literal=password='<new_password>' \
        --from-literal=certificateVerification='<Disabled_or_Enabled>' \
        --dry-run=client -o yaml | oc apply -f -

      All four keys must be present. The cluster etcd Operator rejects secrets with missing keys.

   3. Verify that the new credentials can reach the BMC by running the following command:

      $ sudo pcs stonith config <node_name>_redfish

   4. Verify that no STONITH resources are blocked by running the following command:

      $ sudo pcs status --full

      The cluster etcd Operator performs this validation automatically when it applies credentials from the secret by using the following command:

      $ fence_redfish --action status

      If the cluster recovers after updating the fencing secrets, no further action is required. If the issue persists, proceed to the next step.

2. Recover from a single-node failure:

   1. Gather initial diagnostics by running the following command:

      $ sudo pcs status --full

      This command provides a detailed view of the current cluster and resource states. You can use the output to identify issues with fencing or etcd startup.

   2. Run the following additional diagnostic commands, if necessary:

      Reset the resources on your cluster by running the following command:

      $ sudo pcs resource cleanup

   3. Review all Pacemaker activity on the node by running the following command:

      $ sudo journalctl -u pacemaker

   4. Diagnose etcd resource startup issues by running the following command:

      $ sudo journalctl -u pacemaker | grep podman-etcd

   5. View the fencing configuration for the node by running the following command:

      $ sudo pcs stonith config <node_name>_redfish

      If fencing is required but is not functioning, ensure that the Redfish fencing endpoint is accessible and verify that the credentials are correct.

      If you have verified the failed node is permanently inaccessible but automated fencing cannot function, verify the failed node meets ALL of the following conditions:

      • The node is powered off and cannot be restarted.
      • The node cannot access any shared storage or cluster resources.
      • The node is completely isolated from the cluster network.

   6. Confirm the node is fenced by running the following command:

      $ sudo pcs stonith confirm <failed_node_name>

      Warning

      If the failed node is accessible or can access shared resources, confirming fencing can cause data corruption and cluster failure.

   7. If etcd is not starting despite fencing being operational, restore etcd from a backup by running the following commands:

      $ sudo cp -r /var/lib/etcd-backup/* /var/lib/etcd/

      $ sudo chown -R etcd:etcd /var/lib/etcd

      If the recovery is successful, no further action is required. If the issue persists, proceed to the next step.

3. Recover from dual node power loss where both nodes are recoverable:

   This procedure applies when both control plane nodes lost power and both nodes can be restarted. If only one node can be restarted, proceed to step 4.

   1. Power on both control plane nodes.

      Pacemaker starts automatically and begins the recovery operation when it detects both nodes are online. If the recovery does not start as expected, use the diagnostic commands described in the previous step to investigate the issue.

   2. Reset the resources on your cluster and instruct Pacemaker to attempt to start them fresh by running the following command:

      $ sudo pcs resource cleanup

   3. Check resource start order by running the following command:

      $ sudo pcs status --full

   4. Inspect the pacemaker service journal if kubelet fails by running the following commands:

      $ sudo journalctl -u pacemaker

      $ sudo journalctl -u kubelet

   5. Handle out-of-sync etcd.

      If one node has a more up-to-date etcd, Pacemaker attempts to fence the lagging node and start it as a learner. If this process stalls, verify the Redfish fencing endpoint and credentials by running the following command:

      $ sudo pcs stonith config

      If the recovery is successful, no further action is required. If the issue persists, perform manual recovery as described in the next step.

4. Restore corosync quorum after dual node power loss (single node recoverable):

   This procedure applies when both control plane nodes lost power and only one node can be restarted. In this scenario, the cluster has lost corosync quorum because the last known state showed both nodes were online before the power loss.

   Important

   Perform this procedure only when both of the following conditions are met:

   • Both control plane nodes lost power
   • Only one control plane node can be restarted

   This scenario typically occurs when you need to replace a control plane node (one node is not recoverable) and the surviving node lost power before the replacement procedure.

   1. Verify that only one node is online by running the following command on the surviving node:

      $ sudo pcs status --full

      The output shows only one node online. The sample output is as follows:

      Cluster name: TNF
      Cluster Summary:
        * Stack: corosync (Pacemaker is running)
        * Current DC: NONE
        * Last updated: Wed Apr 29 16:21:17 2026 on master-0.ostest.test.metalkube.org
        * Last change:  Wed Apr 29 16:19:25 2026 by root via root on master-1.ostest.test.metalkube.org
        * 2 nodes configured
        * 6 resource instances configured
      
      Node List:
        * Node master-0.ostest.test.metalkube.org (1): UNCLEAN (offline)
        * Node master-1.ostest.test.metalkube.org (2): UNCLEAN (offline)
      
      Full List of Resources:
        * Clone Set: kubelet-clone [kubelet]:
          * kubelet	(systemd:kubelet):	 Stopped
          * kubelet	(systemd:kubelet):	 Stopped
        * master-0.ostest.test.metalkube.org_redfish	(stonith:fence_redfish):	 Stopped
        * master-1.ostest.test.metalkube.org_redfish	(stonith:fence_redfish):	 Stopped
        * Clone Set: etcd-clone [etcd]:
          * etcd	(ocf:heartbeat:podman-etcd):	 Stopped
          * etcd	(ocf:heartbeat:podman-etcd):	 Stopped
      
      Tickets:
      
      PCSD Status:
        master-0.ostest.test.metalkube.org: Online
        master-1.ostest.test.metalkube.org: Offline
      
      Daemon Status:
        corosync: active/enabled
        pacemaker: active/enabled
        pcsd: active/enabled

      The PCSD status shows that the master-0 node is Online, and the other is offline. BOTH nodes in the node list section are offline because neither has quorum.

      [core@master-0 ~]$   sudo pcs quorum status --debug
      Running: /usr/sbin/corosync-quorumtool -p
      Environment:
        LC_ALL=C
      
      Finished running: /usr/sbin/corosync-quorumtool -p
      Return value: 2
      --Debug Stdout Start--
      Quorum information
      ------------------
      Date:             Wed Apr 29 16:25:55 2026
      Quorum provider:  corosync_votequorum
      Nodes:            1
      Node ID:          1
      Ring ID:          1.e
      Quorate:          No
      
      Votequorum information
      ----------------------
      Expected votes:   2
      Highest expected: 2
      Total votes:      1
      Quorum:           1 Activity blocked
      Flags:            2Node WaitForAll
      
      Membership information
      ----------------------
          Nodeid      Votes    Qdevice Name
               1          1         NR master-0.ostest.test.metalkube.org (local)
      
      --Debug Stdout End--
      --Debug Stderr Start--
      
      --Debug Stderr End--
      
      Error: Unable to get quorum status:

   2. Verify that the failed node is permanently inaccessible before proceeding.

      Before confirming to Pacemaker that the failed node is fenced, you must ensure that the failed node meets ALL of the following conditions:

      • The node is powered off and cannot be restarted
      • The node cannot access any shared storage or cluster resources

      • The node is completely isolated from the cluster network

        If the failed node is accessible or can access shared resources, DO NOT proceed with this step. Confirming fencing for a node that is still active can cause data corruption and cluster failure.

   3. Confirm to Pacemaker that the failed node is fenced by running the following command:

      $ sudo pcs quorum unblock

      The command shows the following sample output:

      WARNING: If node 'master-1' is not powered off or it does have access to shared resources, data corruption and/or cluster failure may occur
      Type 'yes' or 'y' to proceed, anything else to cancel:

      Replace <failed_node_name> with the name of the failed control plane node (for example, control-plane-1).

   4. Verify that quorum is restored by running the following command:

      $ sudo pcs quorum status

      The command shows the following sample output:

      Example output

      Quorum information
      ------------------
      Date:             Fri Oct  3 14:15:31 2025
      Quorum provider:  corosync_votequorum
      Nodes:            1
      Node ID:          1
      Ring ID:          1.16
      Quorate:          Yes
      
      Votequorum information
      ----------------------
      Expected votes:   2
      Highest expected: 2
      Total votes:      1
      Quorum:           1
      Flags:            2Node Quorate

   5. Wait 30 seconds for Pacemaker to process the fencing confirmation and begin recovery.

   6. Verify that etcd is running on the surviving node by running the following command:

      $ sudo pcs resource status etcd

      If etcd is not running, restart it by running the following command:

      $ sudo pcs resource cleanup etcd

      Wait up to 5 minutes for etcd to start. Check the status periodically by running the following command:

      $ sudo pcs resource status etcd

      The command shows that the podman-etcd resource is started. If the container is started successfully, you can see the logs by running the following command:

      $ sudo podman logs etcd

      If the container is not started, you can see the logs by running the following command:

      $ journalctl -u pacemaker | grep podman-etcd

      The relevant logs are placed at /var/log/paceamaker/pacemaker.log. The output must show that etcd is started on the surviving node.

      After restoring corosync quorum and confirming etcd is running, proceed to step 5 to replace the failed control plane node.

5. If you need to manually recover from an event when one of the nodes is not recoverable, follow the procedure in "Replacing control plane nodes in a two-node OpenShift cluster".

   When a cluster loses a single node, it enters degraded mode. In this state, Pacemaker automatically unblocks quorum and allows the cluster to temporarily operate on the remaining node.

   If both nodes fail and both can be restarted, Pacemaker reestablishes quorum automatically when both nodes are online.

   If only one node can be restarted, proceed to step 4 to restore corosync quorum manually.

   If manual recovery is still required and it fails, collect a must-gather and sosreport, and file a bug.

Procedure

1. Check the quorum state by running the following command:

   $ sudo pcs quorum status

   Example output

   Quorum information
   ------------------
   Date:             Fri Oct  3 14:15:31 2025
   Quorum provider:  corosync_votequorum
   Nodes:            2
   Node ID:          1
   Ring ID:          1.16
   Quorate:          Yes
   
   Votequorum information
   ----------------------
   Expected votes:   2
   Highest expected: 2
   Total votes:      2
   Quorum:           1
   Flags:            2Node Quorate WaitForAll
   
   Membership information
   ----------------------
       Nodeid      Votes    Qdevice Name
            1          1         NR master-0 (local)
            2          1         NR master-1

   1. If quorum is lost and one control plane node is still running, restore quorum manually on the survivor node by running the following command:

      $ sudo pcs quorum unblock

   2. If only one node failed, verify that etcd is running on the survivor node by running the following command:

      $ sudo pcs resource status etcd

   3. If etcd is not running, restart etcd by running the following command:

      $ sudo pcs resource cleanup etcd

      If etcd still does not start, force it manually on the survivor node, skipping fencing:

      Important

      Before running this commands, ensure that the node being replaced is inaccessible. Otherwise, you risk etcd corruption.

      $ sudo pcs resource debug-stop etcd

      $ sudo OCF_RESKEY_CRM_meta_notify_start_resource='etcd' pcs resource debug-start etcd

      After recovery, etcd must be running successfully on the survivor node.

2. Delete etcd secrets for the failed node by running the following commands:

   $ oc project openshift-etcd

   $ oc delete secret etcd-peer-<node_name>

   $ oc delete secret etcd-serving-<node_name>

   $ oc delete secret etcd-serving-metrics-<node_name>

   Note

   To replace the failed node, you must delete its etcd secrets first. When etcd is running, it might take some time for the API server to respond to these commands.

3. Delete resources for the failed node:

   1. If you have the BareMetalHost (BMH) objects, list them to identify the host you are replacing by running the following command:

      $ oc get bmh -n openshift-machine-api

   2. Delete the BMH object for the failed node by running the following command:

      $ oc delete bmh/<bmh_name> -n openshift-machine-api

   3. List the Machine objects to identify the object that maps to the node that you are replacing by running the following command:

      $ oc get machines.machine.openshift.io -n openshift-machine-api

   4. Get the label with the machine hash value from the Machine object by running the following command:

      $ oc get machines.machine.openshift.io/<machine_name> -n openshift-machine-api \
        -o jsonpath='Machine hash label: {.metadata.labels.machine\.openshift\.io/cluster-api-cluster}{"\n"}'

      Replace <machine_name> with the name of a Machine object in your cluster. For example, ostest-bfs7w-ctrlplane-0.

      You need this label to provision a new Machine object.

   5. Delete the Machine object for the failed node by running the following command:

      $ oc delete machines.machine.openshift.io/<machine_name>-<failed nodename> -n openshift-machine-api

      Note

      The node object is deleted automatically after deleting the Machine object.

4. Recreate the failed host by using the same name and IP address:

   Important

   You must perform this step only if you are using installer-provisioned infrastructure or the Machine API to create the original node. For information about replacing a failed bare-metal control plane node, see "Replacing an unhealthy etcd member on bare metal".

   1. Remove the BMH and Machine objects. The machine controller automatically deletes the node object.

   2. Provision a new machine by using the following sample configuration:

      Example Machine object configuration

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          metal3.io/BareMetalHost: openshift-machine-api/{bmh_name}
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: {machine_hash_label}
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: {machine_name}
        namespace: openshift-machine-api
      spec:
        authoritativeAPI: MachineAPI
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
              name: master-user-data-managed

      • metadata.annotations.metal3.io/BareMetalHost: Replace {bmh_name} with the name of the BMH object that is associated with the host that you are replacing.
      • labels.machine.openshift.io/cluster-api-cluster: Replace {machine_hash_label} with the label that you fetched from the machine you deleted.
      • metadata.name: Replace {machine_name} with the name of the machine you deleted.

   3. Create the new BMH object and the secret to store the BMC credentials by running the following command:

      cat <<EOF | oc apply -f -
      apiVersion: v1
      kind: Secret
      metadata:
        name: <secret_name>
        namespace: openshift-machine-api
      data:
        password: <password>
        username: <username>
      type: Opaque
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: {bmh_name}
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: disabled
        bmc:
          address: <redfish_url>/{uuid}
          credentialsName: <name>
          disableCertificateVerification: true
        bootMACAddress: {boot_mac_address}
        bootMode: UEFI
        externallyProvisioned: false
        online: true
        rootDeviceHints:
          deviceName: /dev/disk/by-id/scsi-<serial_number>
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api
      EOF

      • metadata.name: Specify the name of the secret.
      • metadata.name: Replace {bmh_name} with the name of the BMH object that you deleted.
      • bmc.address: Replace {uuid} with the UUID of the node that you created.
      • bmc.credentialsName: Replace name with the name of the secret that you created.
      • bootMACAddress: Specify the MAC address of the provisioning network interface. This is the MAC address the node uses to identify itself when communicating with Ironic during provisioning.

5. Verify that the new node has reached the Provisioned state by running the following command:

   $ oc get bmh -o wide

   The value of the STATUS column in the output of this command must be Provisioned.

   Note

   The provisioning process can take 10 to 20 minutes to complete.

6. Verify that both control plane nodes are in the Ready state by running the following command:

   $ oc get nodes

   The value of the STATUS column in the output of this command must be Ready for both nodes.

7. Apply the detached annotation to the BMH object to prevent the Machine API from managing it by running the following command:

   $ oc annotate bmh <bmh_name> -n openshift-machine-api baremetalhost.metal3.io/detached='' --overwrite

8. Rejoin the replacement node to the pacemaker cluster by running the following command:

   Note

   Run the following command on the survivor control plane node, not the node being replaced.

   $ sudo pcs cluster node remove <node_name>

   $ sudo pcs cluster node add <node_name> addr=<node_ip> --start --enable

9. Delete stale jobs for the failed node by running the following command:

   $ oc project openshift-etcd

   $ oc delete job tnf-auth-job-<node_name>

   $ oc delete job tnf-after-setup-job-<node_name>

Procedure

1. Check the overall node status by running the following command:

   $ oc get nodes

   This command verifies that both control plane nodes are in the Ready state, indicating that they can receive workloads for scheduling.

2. Verify the status of the cluster-etcd-operator by running the following command:

   $ oc describe co/etcd

   The cluster-etcd-operator manages and reports on the health of your etcd setup. Reviewing its status helps you identify any ongoing issues or degraded conditions.

3. Review the etcd member list by running the following command:

   $ oc rsh -n openshift-etcd <etcd_pod> etcdctl member list -w table

   This command shows the current etcd members and their roles. Look for any nodes marked as learner, which indicates that they are in the process of becoming voting members.

4. Review the Pacemaker resource status by running the following command on either control plane node:

   $ sudo pcs status --full

   This command provides a detailed overview of all resources managed by Pacemaker. You must ensure that the following conditions are met:

   • Both nodes are online.
   • The kubelet and etcd resources are running.
   • Fencing is correctly configured for both nodes.