Chapter 11. Expanding the cluster

Procedure

1. Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:

   $ oc get csr | grep Pending

   Example output

   csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
   csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:node-5                                                   <none>              Pending

2. Approve all pending CSRs for the new node (node-5 in this example):

   $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

   Important

   You must approve the CSRs to complete the installation.

3. Confirm that the new control plane node is in Ready status:

   $ oc get nodes

   Example output

   NAME       STATUS   ROLES    AGE     VERSION
   node-0   Ready    master   4h42m   v1.24.0+3882f8f
   node-1   Ready    master   4h27m   v1.24.0+3882f8f
   node-2   Ready    master   4h43m   v1.24.0+3882f8f
   node-3   Ready    worker   4h29m   v1.24.0+3882f8f
   node-4   Ready    worker   4h30m   v1.24.0+3882f8f
   node-5   Ready    master   105s    v1.24.0+3882f8f

   Note

   The etcd operator requires a Machine custom resource (CR) that references the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three or more control plane nodes.

4. Create the BareMetalHost and Machine CRs and link them to the new control plane’s Node CR.

   1. Create the BareMetalHost CR with a unique .metadata.name value:

      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: node-5
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: metadata
        bootMACAddress: 00:00:00:00:00:02
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        externallyProvisioned: true
        online: true
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api

   2. Apply the BareMetalHost CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the BareMetalHost CR.

   3. Create the Machine CR using the unique .metadata.name value:

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: externally provisioned
          metal3.io/BareMetalHost: openshift-machine-api/node-5
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: <cluster_name>
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: node-5
        namespace: openshift-machine-api
      spec:
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
              name: master-user-data-managed

      Replace <cluster_name> with the name of the specific cluster, for example, test-day2-1-6qv96.

      To get the cluster name, run the following command:

      $ oc get infrastructure cluster -o=jsonpath='{.status.infrastructureName}{"\n"}'

   4. Apply the Machine CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the Machine CR.

   5. Link BareMetalHost, Machine, and Node by running the link-machine-and-node.sh script:

      1. Copy the link-machine-and-node.sh script below to a local machine:

         #!/bin/bash
         
         # Credit goes to
         # https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
         # This script will link Machine object
         # and Node object. This is needed
         # in order to have IP address of
         # the Node present in the status of the Machine.
         
         set -e
         
         machine="$1"
         node="$2"
         
         if [ -z "$machine" ] || [ -z "$node" ]; then
             echo "Usage: $0 MACHINE NODE"
             exit 1
         fi
         
         node_name=$(echo "${node}" | cut -f2 -d':')
         
         oc proxy &
         proxy_pid=$!
         function kill_proxy {
             kill $proxy_pid
         }
         trap kill_proxy EXIT SIGINT
         
         HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
         
         function print_nics() {
             local ips
             local eob
             declare -a ips
         
             readarray -t ips < <(echo "${1}" \
                                  | jq '.[] | select(. | .type == "InternalIP") | .address' \
                                  | sed 's/"//g')
         
             eob=','
             for (( i=0; i<${#ips[@]}; i++ )); do
                 if [ $((i+1)) -eq ${#ips[@]} ]; then
                     eob=""
                 fi
                 cat <<- EOF
                   {
                     "ip": "${ips[$i]}",
                     "mac": "00:00:00:00:00:00",
                     "model": "unknown",
                     "speedGbps": 10,
                     "vlanId": 0,
                     "pxe": true,
                     "name": "eth1"
                   }${eob}
         EOF
             done
         }
         
         function wait_for_json() {
             local name
             local url
             local curl_opts
             local timeout
         
             local start_time
             local curr_time
             local time_diff
         
             name="$1"
             url="$2"
             timeout="$3"
             shift 3
             curl_opts="$@"
             echo -n "Waiting for $name to respond"
             start_time=$(date +%s)
             until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
                 echo -n "."
                 curr_time=$(date +%s)
                 time_diff=$((curr_time - start_time))
                 if [[ $time_diff -gt $timeout ]]; then
                     printf '\nTimed out waiting for %s' "${name}"
                     return 1
                 fi
                 sleep 5
             done
             echo " Success!"
             return 0
         }
         wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
         
         addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses')
         
         machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}")
         host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
         
         if [ -z "$host" ]; then
             echo "Machine $machine is not linked to a host yet." 1>&2
             exit 1
         fi
         
         # The address structure on the host doesn't match the node, so extract
         # the values we want into separate variables so we can build the patch
         # we need.
         hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
         
         set +e
         read -r -d '' host_patch << EOF
         {
           "status": {
             "hardware": {
               "hostname": "${hostname}",
               "nics": [
         $(print_nics "${addresses}")
               ],
               "systemVendor": {
                 "manufacturer": "Red Hat",
                 "productName": "product name",
                 "serialNumber": ""
               },
               "firmware": {
                 "bios": {
                   "date": "04/01/2014",
                   "vendor": "SeaBIOS",
                   "version": "1.11.0-2.el7"
                 }
               },
               "ramMebibytes": 0,
               "storage": [],
               "cpu": {
                 "arch": "x86_64",
                 "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
                 "clockMegahertz": 2199.998,
                 "count": 4,
                 "flags": []
               }
             }
           }
         }
         EOF
         set -e
         
         echo "PATCHING HOST"
         echo "${host_patch}" | jq .
         
         curl -s \
              -X PATCH \
              "${HOST_PROXY_API_PATH}/${host}/status" \
              -H "Content-type: application/merge-patch+json" \
              -d "${host_patch}"
         
         oc get baremetalhost -n openshift-machine-api -o yaml "${host}"

      2. Make the script executable:

         $ chmod +x link-machine-and-node.sh

      3. Run the script:

         $ bash link-machine-and-node.sh node-5 node-5

         Note

         The first node-5 instance represents the machine, and the second represents the node.

5. Confirm members of etcd by executing into one of the pre-existing control plane nodes:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-0

   2. List etcd members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |76ae1d00| started |node-0  |192.168.111.24|192.168.111.24|  false  |
      |2c18942f| started |node-1  |192.168.111.26|192.168.111.26|  false  |
      |61e2a860| started |node-2  |192.168.111.25|192.168.111.25|  false  |
      |ead5f280| started |node-5  |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

6. Monitor the etcd operator configuration process until completion:

   $ oc get clusteroperator etcd

   Example output (upon completion)

   NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   etcd   4.11.5    True        False         False      5h54m

7. Confirm etcd health by running the following commands:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-0

   2. Check endpoint health:

      # etcdctl endpoint health

      Example output

      192.168.111.24 is healthy: committed proposal: took = 10.383651ms
      192.168.111.26 is healthy: committed proposal: took = 11.297561ms
      192.168.111.25 is healthy: committed proposal: took = 13.892416ms
      192.168.111.28 is healthy: committed proposal: took = 11.870755ms

8. Verify that all nodes are ready:

   $ oc get nodes

   Example output

   NAME      STATUS   ROLES    AGE     VERSION
   node-0    Ready    master   6h20m   v1.24.0+3882f8f
   node-1    Ready    master   6h20m   v1.24.0+3882f8f
   node-2    Ready    master   6h4m    v1.24.0+3882f8f
   node-3    Ready    worker   6h7m    v1.24.0+3882f8f
   node-4    Ready    worker   6h7m    v1.24.0+3882f8f
   node-5    Ready    master   99m     v1.24.0+3882f8f

9. Verify that the cluster Operators are all available:

   $ oc get ClusterOperators

   Example output

   NAME                                      VERSION AVAILABLE PROGRESSING DEGRADED SINCE MSG
   authentication                            4.11.5  True      False       False    5h57m
   baremetal                                 4.11.5  True      False       False    6h19m
   cloud-controller-manager                  4.11.5  True      False       False    6h20m
   cloud-credential                          4.11.5  True      False       False    6h23m
   cluster-autoscaler                        4.11.5  True      False       False    6h18m
   config-operator                           4.11.5  True      False       False    6h19m
   console                                   4.11.5  True      False       False    6h4m
   csi-snapshot-controller                   4.11.5  True      False       False    6h19m
   dns                                       4.11.5  True      False       False    6h18m
   etcd                                      4.11.5  True      False       False    6h17m
   image-registry                            4.11.5  True      False       False    6h7m
   ingress                                   4.11.5  True      False       False    6h6m
   insights                                  4.11.5  True      False       False    6h12m
   kube-apiserver                            4.11.5  True      False       False    6h16m
   kube-controller-manager                   4.11.5  True      False       False    6h16m
   kube-scheduler                            4.11.5  True      False       False    6h16m
   kube-storage-version-migrator             4.11.5  True      False       False    6h19m
   machine-api                               4.11.5  True      False       False    6h15m
   machine-approver                          4.11.5  True      False       False    6h19m
   machine-config                            4.11.5  True      False       False    6h18m
   marketplace                               4.11.5  True      False       False    6h18m
   monitoring                                4.11.5  True      False       False    6h4m
   network                                   4.11.5  True      False       False    6h20m
   node-tuning                               4.11.5  True      False       False    6h18m
   openshift-apiserver                       4.11.5  True      False       False    6h8m
   openshift-controller-manager              4.11.5  True      False       False    6h7m
   openshift-samples                         4.11.5  True      False       False    6h12m
   operator-lifecycle-manager                4.11.5  True      False       False    6h18m
   operator-lifecycle-manager-catalog        4.11.5  True      False       False    6h19m
   operator-lifecycle-manager-pkgsvr         4.11.5  True      False       False    6h12m
   service-ca                                4.11.5  True      False       False    6h19m
   storage                                   4.11.5  True      False       False    6h19m

10. Verify that the cluster version is correct:

    $ oc get ClusterVersion

    Example output

    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.11.5    True        False         5h57m   Cluster version is 4.11.5

Procedure

1. Delete the BareMetalHost CR of the pre-existing control plane node:

   $ oc delete bmh -n openshift-machine-api node-0

2. Confirm that the machine is unhealthy:

   $ oc get machine -A

   Example output

   NAMESPACE               NAME     PHASE     AGE
   openshift-machine-api   node-0   Failed    20h
   openshift-machine-api   node-1   Running   20h
   openshift-machine-api   node-2   Running   20h
   openshift-machine-api   node-3   Running   19h
   openshift-machine-api   node-4   Running   19h
   openshift-machine-api   node-5   Running   14h

3. Delete the Machine CR:

   $ oc delete machine -n openshift-machine-api node-0
   machine.machine.openshift.io "node-0" deleted

4. Confirm removal of the Node CR:

   $ oc get nodes

   Example output

   NAME      STATUS   ROLES    AGE   VERSION
   node-1    Ready    master   20h   v1.24.0+3882f8f
   node-2    Ready    master   19h   v1.24.0+3882f8f
   node-3    Ready    worker   19h   v1.24.0+3882f8f
   node-4    Ready    worker   19h   v1.24.0+3882f8f
   node-5    Ready    master   15h   v1.24.0+3882f8f

5. Check etcd-operator logs to confirm status of the etcd cluster:

   $ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf

   Example output

   E0927 07:53:10.597523       1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resource

6. Remove the physical machine to allow the etcd operator to reconcile the cluster members:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-1

   2. Monitor the progress of etcd operator reconciliation by checking members and endpoint health:

      # etcdctl member list -w table; etcdctl endpoint health

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |61e2a860| started | node-2 |192.168.111.25|192.168.111.25|  false  |
      |ead4f280| started | node-5 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+
      192.168.111.26 is healthy: committed proposal: took = 10.458132ms
      192.168.111.25 is healthy: committed proposal: took = 11.047349ms
      192.168.111.28 is healthy: committed proposal: took = 11.414402ms

Procedure

1. Check the node status to confirm that a control plane node is not ready:

   $ oc get nodes

   Example output

   NAME      STATUS      ROLES    AGE   VERSION
   node-0    NotReady    master   20h   v1.24.0+3882f8f
   node-1    Ready       master   20h   v1.24.0+3882f8f
   node-2    Ready       master   20h   v1.24.0+3882f8f
   node-3    Ready       worker   20h   v1.24.0+3882f8f
   node-4    Ready       worker   20h   v1.24.0+3882f8f

2. Confirm in the etcd-operator logs that the cluster is unhealthy:

   $ oc logs -n openshift-etcd-operator deployment/etcd-operator

   Example output

   E0927 08:24:23.983733       1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, node-0 is unhealthy

3. Confirm the etcd members by running the following commands:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1

   2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |61e2a860| started | node-0 |192.168.111.25|192.168.111.25|  false  |
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |ead4f280| started | node-2 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

4. Confirm that etcdctl endpoint health reports an unhealthy member of the cluster:

   # etcdctl endpoint health

   Example output

   {"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
   192.168.111.28 is healthy: committed proposal: took = 12.465641ms
   192.168.111.26 is healthy: committed proposal: took = 12.297059ms
   192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
   Error: unhealthy cluster

5. Remove the unhealthy control plane by deleting the Machine custom resource (CR):

   $ oc delete machine -n openshift-machine-api node-0

   Note

   The Machine and Node CRs might not be deleted because they are protected by finalizers. If this occurs, you must delete the Machine CR manually by removing all finalizers.

6. Verify in the etcd-operator logs whether the unhealthy machine has been removed:

   $ oc logs -n openshift-etcd-operator deployment/etcd-operator

   Example output

   I0927 08:58:41.249222       1 machinedeletionhooks.go:135] skip removing the deletion hook from machine node-0 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.25}]

7. If you see that removal has been skipped, as in the above log example, manually remove the unhealthy etcdctl member:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1

   2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |61e2a860| started | node-0 |192.168.111.25|192.168.111.25|  false  |
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |ead4f280| started | node-2 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

   3. Confirm that etcdctl endpoint health reports an unhealthy member of the cluster:

      # etcdctl endpoint health

      Example output

      {"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
      192.168.111.28 is healthy: committed proposal: took = 13.038278ms
      192.168.111.26 is healthy: committed proposal: took = 12.950355ms
      192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
      Error: unhealthy cluster

   4. Remove the unhealthy etcdctl member from the cluster:

      # etcdctl member remove 61e2a86084aafa62

      Example output

      Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7

   5. Verify that the unhealthy etcdctl member was removed by running the following command:

      # etcdctl member list -w table

      Example output

      +----------+---------+--------+--------------+--------------+-------+
      |    ID    | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS |LEARNER|
      +----------+---------+--------+--------------+--------------+-------+
      | 2c18942f | started | node-1 |192.168.111.26|192.168.111.26| false |
      | ead4f280 | started | node-2 |192.168.111.28|192.168.111.28| false |
      +----------+---------+--------+--------------+--------------+-------+

Procedure

1. Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:

   $ oc get csr | grep Pending

   Example output

   csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
   csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:node-5                                                   <none>              Pending

2. Approve all pending CSRs for the new node (node-5 in this example):

   $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

   Note

   You must approve the CSRs to complete the installation.

3. Confirm that the control plane node is in Ready status:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES     AGE     VERSION
   node-1    Ready     master    20h     v1.24.0+3882f8f
   node-2    Ready     master    20h     v1.24.0+3882f8f
   node-3    Ready     worker    20h     v1.24.0+3882f8f
   node-4    Ready     worker    20h     v1.24.0+3882f8f
   node-5    Ready     master    2m52s   v1.24.0+3882f8f

   The etcd operator requires a Machine CR referencing the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three control plane nodes.

4. Create the BareMetalHost and Machine CRs and link them to the new control plane’s Node CR.

   Important

   Boot-it-yourself will not create BareMetalHost and Machine CRs, so you must create them. Failure to create the BareMetalHost and Machine CRs will generate errors in the etcd operator.

   1. Create the BareMetalHost CR with a unique .metadata.name value:

      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: node-5
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: metadata
        bootMACAddress: 00:00:00:00:00:02
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        externallyProvisioned: true
        online: true
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api

   2. Apply the BareMetalHost CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the BareMetalHost CR.

   3. Create the Machine CR using the unique .metadata.name value:

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: externally provisioned
          metal3.io/BareMetalHost: openshift-machine-api/node-5
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: test-day2-1-6qv96
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: node-5
        namespace: openshift-machine-api
      spec:
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
             name: master-user-data-managed

   4. Apply the Machine CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the Machine CR.

   5. Link BareMetalHost, Machine, and Node by running the link-machine-and-node.sh script:

      1. Copy the link-machine-and-node.sh script below to a local machine:

         #!/bin/bash
         
         # Credit goes to
         # https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
         # This script will link Machine object
         # and Node object. This is needed
         # in order to have IP address of
         # the Node present in the status of the Machine.
         
         set -e
         
         machine="$1"
         node="$2"
         
         if [ -z "$machine" ] || [ -z "$node" ]; then
             echo "Usage: $0 MACHINE NODE"
             exit 1
         fi
         
         node_name=$(echo "${node}" | cut -f2 -d':')
         
         oc proxy &
         proxy_pid=$!
         function kill_proxy {
             kill $proxy_pid
         }
         trap kill_proxy EXIT SIGINT
         
         HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
         
         function print_nics() {
             local ips
             local eob
             declare -a ips
         
             readarray -t ips < <(echo "${1}" \
                                  | jq '.[] | select(. | .type == "InternalIP") | .address' \
                                  | sed 's/"//g')
         
             eob=','
             for (( i=0; i<${#ips[@]}; i++ )); do
                 if [ $((i+1)) -eq ${#ips[@]} ]; then
                     eob=""
                 fi
                 cat <<- EOF
                   {
                     "ip": "${ips[$i]}",
                     "mac": "00:00:00:00:00:00",
                     "model": "unknown",
                     "speedGbps": 10,
                     "vlanId": 0,
                     "pxe": true,
                     "name": "eth1"
                   }${eob}
         EOF
             done
         }
         
         function wait_for_json() {
             local name
             local url
             local curl_opts
             local timeout
         
             local start_time
             local curr_time
             local time_diff
         
             name="$1"
             url="$2"
             timeout="$3"
             shift 3
             curl_opts="$@"
             echo -n "Waiting for $name to respond"
             start_time=$(date +%s)
             until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
                 echo -n "."
                 curr_time=$(date +%s)
                 time_diff=$((curr_time - start_time))
                 if [[ $time_diff -gt $timeout ]]; then
                     printf '\nTimed out waiting for %s' "${name}"
                     return 1
                 fi
                 sleep 5
             done
             echo " Success!"
             return 0
         }
         wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
         
         addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses')
         
         machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}")
         host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
         
         if [ -z "$host" ]; then
             echo "Machine $machine is not linked to a host yet." 1>&2
             exit 1
         fi
         
         # The address structure on the host doesn't match the node, so extract
         # the values we want into separate variables so we can build the patch
         # we need.
         hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
         
         set +e
         read -r -d '' host_patch << EOF
         {
           "status": {
             "hardware": {
               "hostname": "${hostname}",
               "nics": [
         $(print_nics "${addresses}")
               ],
               "systemVendor": {
                 "manufacturer": "Red Hat",
                 "productName": "product name",
                 "serialNumber": ""
               },
               "firmware": {
                 "bios": {
                   "date": "04/01/2014",
                   "vendor": "SeaBIOS",
                   "version": "1.11.0-2.el7"
                 }
               },
               "ramMebibytes": 0,
               "storage": [],
               "cpu": {
                 "arch": "x86_64",
                 "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
                 "clockMegahertz": 2199.998,
                 "count": 4,
                 "flags": []
               }
             }
           }
         }
         EOF
         set -e
         
         echo "PATCHING HOST"
         echo "${host_patch}" | jq .
         
         curl -s \
              -X PATCH \
              "${HOST_PROXY_API_PATH}/${host}/status" \
              -H "Content-type: application/merge-patch+json" \
              -d "${host_patch}"
         
         oc get baremetalhost -n openshift-machine-api -o yaml "${host}"

      2. Make the script executable:

         $ chmod +x link-machine-and-node.sh

      3. Run the script:

         $ bash link-machine-and-node.sh node-5 node-5

         Note

         The first node-5 instance represents the machine, and the second represents the node.

5. Confirm members of etcd by running the following commands:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1

   2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +---------+-------+--------+--------------+--------------+-------+
      |   ID    | STATUS|  NAME  |   PEER ADDRS | CLIENT ADDRS |LEARNER|
      +---------+-------+--------+--------------+--------------+-------+
      | 2c18942f|started| node-1 |192.168.111.26|192.168.111.26| false |
      | ead4f280|started| node-2 |192.168.111.28|192.168.111.28| false |
      | 79153c5a|started| node-5 |192.168.111.29|192.168.111.29| false |
      +---------+-------+--------+--------------+--------------+-------+

6. Monitor the etcd operator configuration process until completion:

   $ oc get clusteroperator etcd

   Example output (upon completion)

   NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   etcd   4.11.5    True        False         False      22h

7. Confirm etcdctl health by running the following commands:

   1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1

   2. Check endpoint health:

      # etcdctl endpoint health

      Example output

      192.168.111.26 is healthy: committed proposal: took = 9.105375ms
      192.168.111.28 is healthy: committed proposal: took = 9.15205ms
      192.168.111.29 is healthy: committed proposal: took = 10.277577ms

8. Confirm the health of the nodes:

   $ oc get Nodes

   Example output

   NAME     STATUS   ROLES    AGE   VERSION
   node-1   Ready    master   20h   v1.24.0+3882f8f
   node-2   Ready    master   20h   v1.24.0+3882f8f
   node-3   Ready    worker   20h   v1.24.0+3882f8f
   node-4   Ready    worker   20h   v1.24.0+3882f8f
   node-5   Ready    master   40m   v1.24.0+3882f8f

9. Verify that the cluster Operators are all available:

   $ oc get ClusterOperators

   Example output

   NAME                               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                     4.11.5    True        False         False      150m
   baremetal                          4.11.5    True        False         False      22h
   cloud-controller-manager           4.11.5    True        False         False      22h
   cloud-credential                   4.11.5    True        False         False      22h
   cluster-autoscaler                 4.11.5    True        False         False      22h
   config-operator                    4.11.5    True        False         False      22h
   console                            4.11.5    True        False         False      145m
   csi-snapshot-controller            4.11.5    True        False         False      22h
   dns                                4.11.5    True        False         False      22h
   etcd                               4.11.5    True        False         False      22h
   image-registry                     4.11.5    True        False         False      22h
   ingress                            4.11.5    True        False         False      22h
   insights                           4.11.5    True        False         False      22h
   kube-apiserver                     4.11.5    True        False         False      22h
   kube-controller-manager            4.11.5    True        False         False      22h
   kube-scheduler                     4.11.5    True        False         False      22h
   kube-storage-version-migrator      4.11.5    True        False         False      148m
   machine-api                        4.11.5    True        False         False      22h
   machine-approver                   4.11.5    True        False         False      22h
   machine-config                     4.11.5    True        False         False      110m
   marketplace                        4.11.5    True        False         False      22h
   monitoring                         4.11.5    True        False         False      22h
   network                            4.11.5    True        False         False      22h
   node-tuning                        4.11.5    True        False         False      22h
   openshift-apiserver                4.11.5    True        False         False      163m
   openshift-controller-manager       4.11.5    True        False         False      22h
   openshift-samples                  4.11.5    True        False         False      22h
   operator-lifecycle-manager         4.11.5    True        False         False      22h
   operator-lifecycle-manager-catalog 4.11.5    True        False         False      22h
   operator-lifecycle-manager-pkgsvr  4.11.5    True        False         False      22h
   service-ca                         4.11.5    True        False         False      22h
   storage                            4.11.5    True        False         False      22h

10. Verify that the cluster version is correct:

    $ oc get ClusterVersion

    Example output

    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.11.5    True        False         22h     Cluster version is 4.11.5