Deploying installer-provisioned clusters on bare metal

1. One CPU is equivalent to one physical core when simultaneous multithreading (SMT), or Hyper-Threading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = CPUs.
2. OpenShift Container Platform and Kubernetes are sensitive to disk performance, and faster storage is recommended, particularly for etcd on the control plane nodes. Note that on many cloud platforms, storage size and IOPS scale together, so you might need to over-allocate storage volume to obtain sufficient performance.

Procedure

1. Boot the node and enter the BIOS menu.
2. Set the node’s boot mode to UEFI Enabled.
3. Enable Secure Boot.

Procedure

1. Log in to the provisioner node via ssh.

2. Create a non-root user (kni) and provide that user with sudo privileges:

   # useradd kni

   # passwd kni

   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni

   # chmod 0440 /etc/sudoers.d/kni

3. Create an ssh key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

4. Log in as the new user on the provisioner node:

   # su - kni

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach

   $ sudo subscription-manager repos --enable=rhel-9-for-<architecture>-appstream-rpms --enable=rhel-9-for-<architecture>-baseos-rpms

   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

7. Modify the user to add the libvirt group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

8. Restart firewalld and enable the http service:

   $ sudo systemctl start firewalld

   $ sudo firewall-cmd --zone=public --add-service=http --permanent

   $ sudo firewall-cmd --reload

9. Start and enable the libvirtd service:

   $ sudo systemctl enable libvirtd --now

10. Create the default storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images

    $ sudo virsh pool-start default

    $ sudo virsh pool-autostart default

11. Create a pull-secret.txt file:

    $ vim pull-secret.txt

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Procedure

1. Log in to the node by using the ssh command.

2. View the NTP servers available to the node by running the following command:

   $ chronyc sources

   Example output

   MS Name/IP address         Stratum Poll Reach LastRx Last sample
   ===============================================================================
   ^+ time.cloudflare.com           3  10   377   187   -209us[ -209us] +/-   32ms
   ^+ t1.time.ir2.yahoo.com         2  10   377   185  -4382us[-4382us] +/-   23ms
   ^+ time.cloudflare.com           3  10   377   198   -996us[-1220us] +/-   33ms
   ^* brenbox.westnet.ie            1  10   377   193  -9538us[-9761us] +/-   24ms

3. Use the ping command to ensure that the node can access an NTP server, for example:

   $ ping time.cloudflare.com

   Example output

   PING time.cloudflare.com (162.159.200.123) 56(84) bytes of data.
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=1 ttl=54 time=32.3 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=2 ttl=54 time=30.9 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=3 ttl=54 time=36.7 ms
   ...

Procedure

1. Export the bare-metal network NIC name by running the following command:

   $ export PUB_CONN=<baremetal_nic_name>

2. Configure the bare-metal network:

   Note

   The SSH connection might disconnect after executing these steps.

   1. For a network using DHCP, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge <con_name> baremetal bridge.stp no 1
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          pkill dhclient;dhclient baremetal
      "

      1

      Replace <con_name> with the connection name.

   2. For a network using static IP addressing and no DHCP network, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge con-name baremetal bridge.stp no ipv4.method manual ipv4.addr "x.x.x.x/yy" ipv4.gateway "a.a.a.a" ipv4.dns "b.b.b.b" 1
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          nmcli con up baremetal
      "

      1

      Replace <con_name> with the connection name. Replace x.x.x.x/yy with the IP address and CIDR for the network. Replace a.a.a.a with the network gateway. Replace b.b.b.b with the IP address of the DNS server.

3. Optional: If you are deploying with a provisioning network, export the provisioning network NIC name by running the following command:

   $ export PROV_CONN=<prov_nic_name>

4. Optional: If you are deploying with a provisioning network, configure the provisioning network by running the following command:

   $ sudo nohup bash -c "
       nmcli con down \"$PROV_CONN\"
       nmcli con delete \"$PROV_CONN\"
       nmcli connection add ifname provisioning type bridge con-name provisioning
       nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
       nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
       nmcli con down provisioning
       nmcli con up provisioning
   "

   Note

   The SSH connection might disconnect after executing these steps.

   The IPv6 address can be any address that is not routable through the bare-metal network.

   Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

5. Optional: If you are deploying with a provisioning network, configure the IPv4 address on the provisioning network connection by running the following command:

   $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

6. SSH back into the provisioner node (if required) by running the following command:

   # ssh kni@provisioner.<cluster-name>.<domain>

7. Verify that the connection bridges have been properly created by running the following command:

   $ sudo nmcli con show

   Example output

   NAME               UUID                                  TYPE      DEVICE
   baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
   provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
   virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
   bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
   bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

Procedure

1. Configure the first subnet to communicate with the second subnet:

   1. Log in as root to a control plane node by running the following command:

      $ sudo su -

   2. Get the name of the network interface by running the following command:

      # nmcli dev status

   3. Add a route to the second subnet (192.168.0.0) via the gateway by running the following command:

      # nmcli connection modify <interface_name> +ipv4.routes "192.168.0.0/24 via <gateway>"

      Replace <interface_name> with the interface name. Replace <gateway> with the IP address of the actual gateway.

      Example

      # nmcli connection modify eth0 +ipv4.routes "192.168.0.0/24 via 192.168.0.1"

   4. Apply the changes by running the following command:

      # nmcli connection up <interface_name>

      Replace <interface_name> with the interface name.

   5. Verify the routing table to ensure the route has been added successfully:

      # ip route

   6. Repeat the previous steps for each control plane node in the first subnet.

      Note

      Adjust the commands to match your actual interface names and gateway.

2. Configure the second subnet to communicate with the first subnet:

   1. Log in as root to a remote compute node by running the following command:

      $ sudo su -

   2. Get the name of the network interface by running the following command:

      # nmcli dev status

   3. Add a route to the first subnet (10.0.0.0) via the gateway by running the following command:

      # nmcli connection modify <interface_name> +ipv4.routes "10.0.0.0/24 via <gateway>"

      Replace <interface_name> with the interface name. Replace <gateway> with the IP address of the actual gateway.

      Example

      # nmcli connection modify eth0 +ipv4.routes "10.0.0.0/24 via 10.0.0.1"

   4. Apply the changes by running the following command:

      # nmcli connection up <interface_name>

      Replace <interface_name> with the interface name.

   5. Verify the routing table to ensure the route has been added successfully by running the following command:

      # ip route

   6. Repeat the previous steps for each compute node in the second subnet.

      Note

      Adjust the commands to match your actual interface names and gateway.

3. After you have configured the networks, test the connectivity to ensure the remote nodes can reach the control plane nodes and the control plane nodes can reach the remote nodes.

   1. From the control plane nodes in the first subnet, ping a remote node in the second subnet by running the following command:

      $ ping <remote_node_ip_address>

      If the ping is successful, it means the control plane nodes in the first subnet can reach the remote nodes in the second subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.

   2. From the remote nodes in the second subnet, ping a control plane node in the first subnet by running the following command:

      $ ping <control_plane_node_ip_address>

      If the ping is successful, it means the remote compute nodes in the second subnet can reach the control plane in the first subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install

   $ export pullsecret_file=~/pull-secret.txt

   $ export extract_dir=$(pwd)

2. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

3. Extract the installer:

   $ sudo cp oc /usr/local/bin

   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}

   $ sudo cp openshift-baremetal-install /usr/local/bin

Procedure

1. Install podman:

   $ sudo dnf install -y podman

2. Open firewall port 8080 to be used for RHCOS image caching:

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   $ sudo firewall-cmd --reload

3. Create a directory to store the bootstraposimage:

   $ mkdir /home/kni/rhcos_image_cache

4. Set the appropriate SELinux context for the newly created directory:

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"

   $ sudo restorecon -Rv /home/kni/rhcos_image_cache/

5. Get the URI for the RHCOS image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_URI=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk.location')

6. Get the name of the image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_NAME=${RHCOS_QEMU_URI##*/}

7. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM:

   $ export RHCOS_QEMU_UNCOMPRESSED_SHA256=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk["uncompressed-sha256"]')

8. Download the image and place it in the /home/kni/rhcos_image_cache directory:

   $ curl -L ${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_NAME}

9. Confirm SELinux type is of httpd_sys_content_t for the new file:

   $ ls -Z /home/kni/rhcos_image_cache

10. Create the pod:

    $ podman run -d --name rhcos_image_cache \1
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    registry.access.redhat.com/ubi9/httpd-24

    1

    Creates a caching webserver with the name rhcos_image_cache. This pod serves the bootstrapOSImage image in the install-config.yaml file for deployment.

11. Generate the bootstrapOSImage configuration:

    $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)

    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_NAME}?sha256=${RHCOS_QEMU_UNCOMPRESSED_SHA256}"

    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"

12. Add the required configuration to the install-config.yaml file under platform.baremetal:

    platform:
      baremetal:
        bootstrapOSImage: <bootstrap_os_image>  1

    1

    Replace <bootstrap_os_image> with the value of $BOOTSTRAP_OS_IMAGE.

    See the "Configuring the install-config.yaml file" section for additional details.

Procedure

1. Power off all bare-metal nodes before installing the OpenShift Container Platform cluster by using the following command:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off

2. Remove all old bootstrap resources if any remain from an earlier deployment attempt by using the following script:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Delete the artifacts that the earlier installation generated by using the following command:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

4. Re-create the OpenShift Container Platform manifests by using the following command:

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

Procedure

1. Check the network interface configuration on the node.
2. Turn off the DHCP server and reboot the OpenShift Container Platform node and ensure that the network configuration works properly.

Procedure

1. Use the guidelines in YAML-tips.
2. Verify the YAML syntax is correct using syntax-check.

3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the install-config.yaml. For example:

   $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.<architecture>.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7

   If the output is 200, there is a valid response from the webserver storing the bootstrap VM image.

Procedure

1. About 10 to 15 minutes after triggering the installation program, check to ensure the bootstrap VM is operational using the virsh command:

   $ sudo virsh list

    Id    Name                           State
    --------------------------------------------
    12    openshift-xf6fq-bootstrap      running

   Note

   The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap."

2. If the bootstrap VM is not running after 10-15 minutes, verify libvirtd is running on the system by executing the following command:

   $ systemctl status libvirtd

   ● libvirtd.service - Virtualization daemon
      Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
        Docs: man:libvirtd(8)
              https://libvirt.org
    Main PID: 9850 (libvirtd)
       Tasks: 20 (limit: 32768)
      Memory: 74.8M
      CGroup: /system.slice/libvirtd.service
              ├─ 9850 /usr/sbin/libvirtd

   If the bootstrap VM is operational, log in to it.

3. Use the virsh console command to find the IP address of the bootstrap VM:

   $ sudo virsh console example.com

   Connected to domain example.com
   Escape character is ^]
   Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
   SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
   SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
   SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
   ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
   ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
   localhost login:

   Important

   When deploying an OpenShift Container Platform cluster without the provisioning network, you must use a public IP address and not a private IP address like 172.22.0.2.

4. After you obtain the IP address, log in to the bootstrap VM using the ssh command:

   Note

   In the console output of the previous step, you can use the IPv6 IP address provided by ens3 or the IPv4 IP provided by ens4.

   $ ssh core@172.22.0.2

Procedure

1. Ensure that etcd is running on each of the control plane nodes by running the following command:

   $ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)

2. If the previous command fails, ensure that Kubelet created the etcd pods by running the following command:

   $ sudo crictl pods --name=etcd-member

   If there are no pods, investigate etcd.

3. Check the cluster nodes to ensure they have a fully qualified domain name, and not just localhost.localdomain, by using the following command:

   $ hostname

   If a hostname is not set, set the correct hostname. For example:

   $ sudo hostnamectl set-hostname <hostname>

4. Ensure that each node has the correct name resolution in the DNS server using the dig command:

   $ dig api.<cluster_name>.example.com

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster_name>.example.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
   ;; QUESTION SECTION:
   ;api.<cluster_name>.example.com. IN A
   
   ;; ANSWER SECTION:
   api.<cluster_name>.example.com. 10800 IN	A 10.19.13.86
   
   ;; AUTHORITY SECTION:
   <cluster_name>.example.com. 10800 IN NS	<cluster_name>.example.com.
   
   ;; ADDITIONAL SECTION:
   <cluster_name>.example.com. 10800 IN A	10.19.14.247
   
   ;; Query time: 0 msec
   ;; SERVER: 10.19.14.247#53(10.19.14.247)
   ;; WHEN: Tue May 19 20:30:59 UTC 2020
   ;; MSG SIZE  rcvd: 140

   The output in the foregoing example indicates that the appropriate IP address for the api.<cluster_name>.example.com VIP is 10.19.13.86. This IP address should reside on the baremetal network.

Procedure

1. Inspect the ClusterVersion object by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion -o yaml

   Example output

   apiVersion: config.openshift.io/v1
   kind: ClusterVersion
   metadata:
     creationTimestamp: 2019-02-27T22:24:21Z
     generation: 1
     name: version
     resourceVersion: "19927"
     selfLink: /apis/config.openshift.io/v1/clusterversions/version
     uid: 6e0f4cf8-3ade-11e9-9034-0a923b47ded4
   spec:
     channel: stable-4.1
     clusterID: 5ec312f9-f729-429d-a454-61d4906896ca
   status:
     availableUpdates: null
     conditions:
     - lastTransitionTime: 2019-02-27T22:50:30Z
       message: Done applying 4.1.1
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:50:30Z
       status: "False"
       type: Failing
     - lastTransitionTime: 2019-02-27T22:50:30Z
       message: Cluster version is 4.1.1
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:24:31Z
       message: 'Unable to retrieve available updates: unknown version 4.1.1
       reason: RemoteFailed
       status: "False"
       type: RetrievedUpdates
     desired:
       image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
       version: 4.1.1
     history:
     - completionTime: 2019-02-27T22:50:30Z
       image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
       startedTime: 2019-02-27T22:24:31Z
       state: Completed
       version: 4.1.1
     observedGeneration: 1
     versionHash: Wa7as_ik1qE=

2. View the conditions by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion version \
        -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

   Some of most important conditions include Failing, Available and Progressing.

   Example output

   Available True Done applying 4.1.1
   Failing False
   Progressing False Cluster version is 4.0.0-0.alpha-2019-02-26-194020
   RetrievedUpdates False Unable to retrieve available updates: unknown version 4.1.1

3. Inspect the ClusterOperator object by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator

   The command returns the status of the cluster Operators.

   Example output

   NAME                                  VERSION   AVAILABLE   PROGRESSING   FAILING   SINCE
   cluster-baremetal-operator                      True        False         False     17m
   cluster-autoscaler                              True        False         False     17m
   cluster-storage-operator                        True        False         False     10m
   console                                         True        False         False     7m21s
   dns                                             True        False         False     31m
   image-registry                                  True        False         False     9m58s
   ingress                                         True        False         False     10m
   kube-apiserver                                  True        False         False     28m
   kube-controller-manager                         True        False         False     21m
   kube-scheduler                                  True        False         False     25m
   machine-api                                     True        False         False     17m
   machine-config                                  True        False         False     17m
   marketplace-operator                            True        False         False     10m
   monitoring                                      True        False         False     8m23s
   network                                         True        False         False     13m
   node-tuning                                     True        False         False     11m
   openshift-apiserver                             True        False         False     15m
   openshift-authentication                        True        False         False     20m
   openshift-cloud-credential-operator             True        False         False     18m
   openshift-controller-manager                    True        False         False     10m
   openshift-samples                               True        False         False     8m42s
   operator-lifecycle-manager                      True        False         False     17m
   service-ca                                      True        False         False     30m

4. Inspect individual cluster Operators by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> -oyaml 1

   1

   Replace <operator> with the name of a cluster Operator. This command is useful for identifying why an cluster Operator has not achieved the Available state or is in the Failed state.

   Example output

   apiVersion: config.openshift.io/v1
   kind: ClusterOperator
   metadata:
     creationTimestamp: 2019-02-27T22:47:04Z
     generation: 1
     name: monitoring
     resourceVersion: "24677"
     selfLink: /apis/config.openshift.io/v1/clusteroperators/monitoring
     uid: 9a6a5ef9-3ae1-11e9-bad4-0a97b6ba9358
   spec: {}
   status:
     conditions:
     - lastTransitionTime: 2019-02-27T22:49:10Z
       message: Successfully rolled out the stack.
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:49:10Z
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:49:10Z
       status: "False"
       type: Failing
     extension: null
     relatedObjects: null
     version: ""

5. To get the cluster Operator’s status condition, run the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> \
        -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

   Replace <operator> with the name of one of the operators above.

   Example output

   Available True Successfully rolled out the stack
   Progressing False
   Failing False

6. To retrieve the list of objects owned by the cluster Operator, execute the following command:

   oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator kube-apiserver \
      -o=jsonpath='{.status.relatedObjects}'

   Example output

   [map[resource:kubeapiservers group:operator.openshift.io name:cluster] map[group: name:openshift-config resource:namespaces] map[group: name:openshift-config-managed resource:namespaces] map[group: name:openshift-kube-apiserver-operator resource:namespaces] map[group: name:openshift-kube-apiserver resource:namespaces]]

Procedure

1. Check if the console router is in the Available or Failing state by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator console -oyaml

   apiVersion: config.openshift.io/v1
   kind: ClusterOperator
   metadata:
     creationTimestamp: 2019-02-27T22:46:57Z
     generation: 1
     name: console
     resourceVersion: "19682"
     selfLink: /apis/config.openshift.io/v1/clusteroperators/console
     uid: 960364aa-3ae1-11e9-bad4-0a97b6ba9358
   spec: {}
   status:
     conditions:
     - lastTransitionTime: 2019-02-27T22:46:58Z
       status: "False"
       type: Failing
     - lastTransitionTime: 2019-02-27T22:50:12Z
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:50:12Z
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:46:57Z
       status: "True"
       type: Upgradeable
     extension: null
     relatedObjects:
     - group: operator.openshift.io
       name: cluster
       resource: consoles
     - group: config.openshift.io
       name: cluster
       resource: consoles
     - group: oauth.openshift.io
       name: console
       resource: oauthclients
     - group: ""
       name: openshift-console-operator
       resource: namespaces
     - group: ""
       name: openshift-console
       resource: namespaces
     versions: null

2. Manually retrieve the console URL by executing the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get route console -n openshift-console \
        -o=jsonpath='{.spec.host}' console-openshift-console.apps.adahiya-1.devcluster.openshift.com

Procedure

1. Retrieve the certificate from the cluster using the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get configmaps default-ingress-cert \
        -n openshift-config-managed -o=jsonpath='{.data.ca-bundle\.crt}'

   -----BEGIN CERTIFICATE-----
   MIIC/TCCAeWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMSwwKgYDVQQDDCNjbHVz
   dGVyLWluZ3Jlc3Mtb3BlcmF0b3JAMTU1MTMwNzU4OTAeFw0xOTAyMjcyMjQ2Mjha
   Fw0yMTAyMjYyMjQ2MjlaMC4xLDAqBgNVBAMMI2NsdXN0ZXItaW5ncmVzcy1vcGVy
   YXRvckAxNTUxMzA3NTg5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
   uCA4fQ+2YXoXSUL4h/mcvJfrgpBfKBW5hfB8NcgXeCYiQPnCKblH1sEQnI3VC5Pk
   2OfNCF3PUlfm4i8CHC95a7nCkRjmJNg1gVrWCvS/ohLgnO0BvszSiRLxIpuo3C4S
   EVqqvxValHcbdAXWgZLQoYZXV7RMz8yZjl5CfhDaaItyBFj3GtIJkXgUwp/5sUfI
   LDXW8MM6AXfuG+kweLdLCMm3g8WLLfLBLvVBKB+4IhIH7ll0buOz04RKhnYN+Ebw
   tcvFi55vwuUCWMnGhWHGEQ8sWm/wLnNlOwsUz7S1/sW8nj87GFHzgkaVM9EOnoNI
   gKhMBK9ItNzjrP6dgiKBCQIDAQABoyYwJDAOBgNVHQ8BAf8EBAMCAqQwEgYDVR0T
   AQH/BAgwBgEB/wIBADANBgkqhkiG9w0BAQsFAAOCAQEAq+vi0sFKudaZ9aUQMMha
   CeWx9CZvZBblnAWT/61UdpZKpFi4eJ2d33lGcfKwHOi2NP/iSKQBebfG0iNLVVPz
   vwLbSG1i9R9GLdAbnHpPT9UG6fLaDIoKpnKiBfGENfxeiq5vTln2bAgivxrVlyiq
   +MdDXFAWb6V4u2xh6RChI7akNsS3oU9PZ9YOs5e8vJp2YAEphht05X0swA+X8V8T
   C278FFifpo0h3Q0Dbv8Rfn4UpBEtN4KkLeS+JeT+0o2XOsFZp7Uhr9yFIodRsnNo
   H/Uwmab28ocNrGNiEVaVH6eTTQeeZuOdoQzUbClElpVmkrNGY0M42K0PvOQ/e7+y
   AQ==
   -----END CERTIFICATE-----

2. Add the certificate to the client-certificate-authority-data field in the ${INSTALL_DIR}/auth/kubeconfig file.

Procedure

1. If you have physical access to the nodes, check their console output to determine if they have successfully booted. If the nodes are still retrieving their boot configuration, there might be problems with the bootstrap VM .
2. Ensure you have configured the sshKey: '<ssh_pub_key>' setting in the install-config.yaml file, where <ssh_pub_key> is the public key of the kni user on the provisioner node.

Procedure

1. Check the network connectivity to the provisioning network.
2. Ensure PXE is enabled on the NIC for the provisioning network and PXE is disabled for all other NICs.

3. Verify that the install-config.yaml configuration file includes the rootDeviceHints parameter and boot MAC address for the NIC connected to the provisioning network. For example:

   control plane node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

   Worker node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

Procedure

1. Check the Machine API Operator by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
      --namespace=openshift-machine-api get deployments

   If ${INSTALL_DIR} is not set in your environment, replace the value with the name of the installation directory.

   Example output

   NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
   cluster-autoscaler-operator   1/1     1            1           86m
   cluster-baremetal-operator    1/1     1            1           86m
   machine-api-controllers       1/1     1            1           85m
   machine-api-operator          1/1     1            1           86m

2. Check the machine controller logs by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
        --namespace=openshift-machine-api logs deployments/machine-api-controllers \
        --container=machine-controller

Procedure

1. Ensure the network configuration exists by running the following command:

   $ oc get network -o yaml cluster

   If it does not exist, the installation program did not create it. To find out why, run the following command:

   $ openshift-install create manifests

   Review the manifests to determine why the installation program did not create the network configuration.

2. Ensure the network is running by entering the following command:

   $ oc get po -n openshift-network-operator

Procedure

1. Add a DNS A/AAAA or CNAME record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   address=/api-int.<cluster_name>.<base_domain>/<IP_address>
   address=/api-int.mycluster.example.com/192.168.1.10
   address=/api-int.mycluster.example.com/2001:0db8:85a3:0000:0000:8a2e:0370:7334

2. Add a DNS PTR record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   ptr-record=<IP_address>.in-addr.arpa,api-int.<cluster_name>.<base_domain>
   ptr-record=10.1.168.192.in-addr.arpa,api-int.mycluster.example.com

3. Restart the DNS server. For example, when using dnsmasq, execute the following command:

   $ sudo systemctl restart dnsmasq

Procedure

1. Power off all bare-metal nodes before installing the OpenShift Container Platform cluster by using the following command:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off

2. Remove all old bootstrap resources if any remain from an earlier deployment attempt by using the following script:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Delete the artifacts that the earlier installation generated by using the following command:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

4. Re-create the OpenShift Container Platform manifests by using the following command:

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

Procedure

1. Check to ensure authentication is successful:

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.json
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Note

   Example output of the variables used to mirror the install images:

   UPSTREAM_REPO=${RELEASE_IMAGE}
   LOCAL_REG=<registry_FQDN>:<registry_port>
   LOCAL_REPO='ocp4/openshift4'

   The values of RELEASE_IMAGE and VERSION were set during the Retrieving OpenShift Installer step of the Setting up the environment for an OpenShift installation section.

2. After mirroring the registry, confirm that you can access it in your disconnected environment:

   $ curl -k -u <user>:<password> https://registry.example.com:<registry_port>/v2/_catalog
   {"repositories":["<Repo_Name>"]}

Procedure

1. When the OpenShift Container Platform cluster nodes are installed appropriately, the following Ready state is seen within the STATUS column:

   $ oc get nodes

   NAME                   STATUS   ROLES           AGE  VERSION
   master-0.example.com   Ready    master,worker   4h   v1.29.4
   master-1.example.com   Ready    master,worker   4h   v1.29.4
   master-2.example.com   Ready    master,worker   4h   v1.29.4

2. Confirm the installation program deployed all pods successfully. The following command removes any pods that are still running or have completed as part of the output.

   $ oc get pods --all-namespaces | grep -iv running | grep -iv complete

Procedure

1. Install Butane on your installation host by using the following command:

   $ sudo dnf -y install butane

2. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.16.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all compute nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

3. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

4. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the compute nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.16.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

5. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

6. Apply the 99-master-chrony-conf-override.yaml policy to the control plane nodes.

   $ oc apply -f 99-master-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-master-chrony-conf-override created

7. Apply the 99-worker-chrony-conf-override.yaml policy to the compute nodes.

   $ oc apply -f 99-worker-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-worker-chrony-conf-override created

8. Check the status of the applied NTP settings.

   $ oc describe machineconfigpool

Procedure

1. When setting the provisioningInterface setting, first identify the provisioning interface name for the cluster nodes. For example, eth0 or eno1.
2. Enable the Preboot eXecution Environment (PXE) on the provisioning network interface of the cluster nodes.

3. Retrieve the current state of the provisioning network and save it to a provisioning custom resource (CR) file:

   $ oc get provisioning -o yaml > enable-provisioning-nw.yaml

4. Modify the provisioning CR file:

   $ vim ~/enable-provisioning-nw.yaml

   Scroll down to the provisioningNetwork configuration setting and change it from Disabled to Managed. Then, add the provisioningIP, provisioningNetworkCIDR, provisioningDHCPRange, provisioningInterface, and watchAllNameSpaces configuration settings after the provisioningNetwork setting. Provide appropriate values for each setting.

   apiVersion: v1
   items:
   - apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       name: provisioning-configuration
     spec:
       provisioningNetwork: 1
       provisioningIP: 2
       provisioningNetworkCIDR: 3
       provisioningDHCPRange: 4
       provisioningInterface: 5
       watchAllNameSpaces: 6

   1

   The provisioningNetwork is one of Managed, Unmanaged, or Disabled. When set to Managed, Metal3 manages the provisioning network and the CBO deploys the Metal3 pod with a configured DHCP server. When set to Unmanaged, the system administrator configures the DHCP server manually.

   2

   The provisioningIP is the static IP address that the DHCP server and ironic use to provision the network. This static IP address must be within the provisioning subnet, and outside of the DHCP range. If you configure this setting, it must have a valid IP address even if the provisioning network is Disabled. The static IP address is bound to the metal3 pod. If the metal3 pod fails and moves to another server, the static IP address also moves to the new server.

   3

   The Classless Inter-Domain Routing (CIDR) address. If you configure this setting, it must have a valid CIDR address even if the provisioning network is Disabled. For example: 192.168.0.1/24.

   4

   The DHCP range. This setting is only applicable to a Managed provisioning network. Omit this configuration setting if the provisioning network is Disabled. For example: 192.168.0.64, 192.168.0.253.

   5

   The NIC name for the provisioning interface on cluster nodes. The provisioningInterface setting is only applicable to Managed and Unmanaged provisioning networks. Omit the provisioningInterface configuration setting if the provisioning network is Disabled. Omit the provisioningInterface configuration setting to use the bootMACAddress configuration setting instead.

   6

   Set this setting to true if you want metal3 to watch namespaces other than the default openshift-machine-api namespace. The default value is false.

5. Save the changes to the provisioning CR file.

6. Apply the provisioning CR file to the cluster:

   $ oc apply -f enable-provisioning-nw.yaml

Procedure

1. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc

   $ sudo cp oc /usr/local/bin

2. Power off the bare metal node by using the baseboard management controller (BMC), and ensure it is off.

3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the user name and password:

   $ echo -ne "root" | base64

   $ echo -ne "password" | base64

4. Create a configuration file for the bare metal node. Depending on whether you are using a static configuration or a DHCP server, use one of the following example bmh.yaml files, replacing values in the YAML to match your environment:

   $ vim bmh.yaml

   • Static configuration bmh.yaml:

     ---
     apiVersion: v1 1
     kind: Secret
     metadata:
      name: openshift-worker-<num>-network-config-secret 2
      namespace: openshift-machine-api
     type: Opaque
     stringData:
      nmstate: | 3
       interfaces: 4
       - name: <nic1_name> 5
         type: ethernet
         state: up
         ipv4:
           address:
           - ip: <ip_address> 6
             prefix-length: 24
           enabled: true
       dns-resolver:
         config:
           server:
           - <dns_ip_address> 7
       routes:
         config:
         - destination: 0.0.0.0/0
           next-hop-address: <next_hop_ip_address> 8
           next-hop-interface: <next_hop_nic1_name> 9
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret 10
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 11
       password: <base64_of_pwd> 12
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num> 13
       namespace: openshift-machine-api
     spec:
       online: True
       bootMACAddress: <nic1_mac_address> 14
       bmc:
         address: <protocol>://<bmc_url> 15
         credentialsName: openshift-worker-<num>-bmc-secret 16
         disableCertificateVerification: True 17
         username: <bmc_username> 18
         password: <bmc_password> 19
       rootDeviceHints:
         deviceName: <root_device_hint> 20
       preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 21

     1

     To configure the network interface for a newly created node, specify the name of the secret that contains the network configuration. Follow the nmstate syntax to define the network configuration for your node. See "Optional: Configuring host network interfaces in the install-config.yaml file" for details on configuring NMState syntax.

     2 10 13 16

     Replace <num> for the worker number of the bare metal node in the name fields, the credentialsName field, and the preprovisioningNetworkDataName field.

     3

     Add the NMState YAML syntax to configure the host interfaces.

     4

     Optional: If you have configured the network interface with nmstate, and you want to disable an interface, set state: up with the IP addresses set to enabled: false as shown:

     ---
        interfaces:
        - name: <nic_name>
          type: ethernet
          state: up
          ipv4:
            enabled: false
          ipv6:
            enabled: false

     5 6 7 8 9

     Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

     11 12

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     14

     Replace <nic1_mac_address> with the MAC address of the bare metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     15

     Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc_url> with the URL of the bare metal node’s baseboard management controller.

     17

     To skip certificate validation, set disableCertificateVerification to true.

     18 19

     Replace <bmc_username> and <bmc_password> with the string of the BMC user name and password.

     20

     Optional: Replace <root_device_hint> with a device path if you specify a root device hint.

     21

     Optional: If you have configured the network interface for the newly created node, provide the network configuration secret name in the preprovisioningNetworkDataName of the BareMetalHost CR.

   • DHCP configuration bmh.yaml:

     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret 1
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 2
       password: <base64_of_pwd> 3
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num> 4
       namespace: openshift-machine-api
     spec:
       online: True
       bootMACAddress: <nic1_mac_address> 5
       bmc:
         address: <protocol>://<bmc_url> 6
         credentialsName: openshift-worker-<num>-bmc-secret 7
         disableCertificateVerification: True 8
         username: <bmc_username> 9
         password: <bmc_password> 10
       rootDeviceHints:
         deviceName: <root_device_hint> 11
       preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 12

     1 4 7

     Replace <num> for the worker number of the bare metal node in the name fields, the credentialsName field, and the preprovisioningNetworkDataName field.

     2 3

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     5

     Replace <nic1_mac_address> with the MAC address of the bare metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     6

     Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc_url> with the URL of the bare metal node’s baseboard management controller.

     8

     To skip certificate validation, set disableCertificateVerification to true.

     9 10

     Replace <bmc_username> and <bmc_password> with the string of the BMC user name and password.

     11

     Optional: Replace <root_device_hint> with a device path if you specify a root device hint.

     12

     Optional: If you have configured the network interface for the newly created node, provide the network configuration secret name in the preprovisioningNetworkDataName of the BareMetalHost CR.

   Note

   If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See "Diagnosing a host duplicate MAC address" for more information.

5. Create the bare metal node:

   $ oc -n openshift-machine-api create -f bmh.yaml

   Example output

   secret/openshift-worker-<num>-network-config-secret created
   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

   Where <num> will be the worker number.

6. Power up and inspect the bare metal node:

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   Example output

   NAME                    STATE       CONSUMER   ONLINE   ERROR
   openshift-worker-<num>  available              true

   Note

   To allow the worker node to join the cluster, scale the machineset object to the number of the BareMetalHost objects. You can scale nodes either manually or automatically. To scale nodes automatically, use the metal3.io/autoscale-to-hosts annotation for machineset.

Procedure

1. Ensure that the Bare Metal Operator is available:

   $ oc get clusteroperator baremetal

   Example output

   NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   baremetal   4.16   True        False         False      3d15h

2. Remove the old BareMetalHost and Machine objects:

   $ oc delete bmh -n openshift-machine-api <host_name>
   $ oc delete machine -n openshift-machine-api <machine_name>

   Replace <host_name> with the name of the host and <machine_name> with the name of the machine. The machine name appears under the CONSUMER field.

   After you remove the BareMetalHost and Machine objects, then the machine controller automatically deletes the Node object.

3. Create the new BareMetalHost object and the secret to store the BMC credentials:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: control-plane-<num>-bmc-secret 1
     namespace: openshift-machine-api
   data:
     username: <base64_of_uid> 2
     password: <base64_of_pwd> 3
   type: Opaque
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: control-plane-<num> 4
     namespace: openshift-machine-api
   spec:
     automatedCleaningMode: disabled
     bmc:
       address: <protocol>://<bmc_ip> 5
       credentialsName: control-plane-<num>-bmc-secret 6
     bootMACAddress: <NIC1_mac_address> 7
     bootMode: UEFI
     externallyProvisioned: false
     online: true
   EOF

   1 4 6

   Replace <num> for the control plane number of the bare metal node in the name fields and the credentialsName field.

   2

   Replace <base64_of_uid> with the base64 string of the user name.

   3

   Replace <base64_of_pwd> with the base64 string of the password.

   5

   Replace <protocol> with the BMC protocol, such as redfish, redfish-virtualmedia, idrac-virtualmedia, or others. Replace <bmc_ip> with the IP address of the bare metal node’s baseboard management controller. For additional BMC configuration options, see "BMC addressing" in the Additional resources section.

   7

   Replace <NIC1_mac_address> with the MAC address of the bare metal node’s first NIC.

   After the inspection is complete, the BareMetalHost object is created and available to be provisioned.

4. View available BareMetalHost objects:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   available                control-plane-1            true             1h10m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             4h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             4h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             4h53m
   compute-1.example.com         provisioned              compute-2-l2zmb            true             4h53m

   There are no MachineSet objects for control plane nodes, so you must create a Machine object instead. You can copy the providerSpec from another control plane Machine object.

5. Create a Machine object:

   $ cat <<EOF | oc apply -f -
   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     annotations:
       metal3.io/BareMetalHost: openshift-machine-api/control-plane-<num> 1
     labels:
       machine.openshift.io/cluster-api-cluster: control-plane-<num> 2
       machine.openshift.io/cluster-api-machine-role: master
       machine.openshift.io/cluster-api-machine-type: master
     name: control-plane-<num> 3
     namespace: openshift-machine-api
   spec:
     metadata: {}
     providerSpec:
       value:
         apiVersion: baremetal.cluster.k8s.io/v1alpha1
         customDeploy:
           method: install_coreos
         hostSelector: {}
         image:
           checksum: ""
           url: ""
         kind: BareMetalMachineProviderSpec
         metadata:
           creationTimestamp: null
         userData:
           name: master-user-data-managed
   EOF

   1 2 3

   Replace <num> for the control plane number of the bare metal node in the name, labels and annotations fields.

6. To view the BareMetalHost objects, run the following command:

   $ oc get bmh -A

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   provisioned              control-plane-1            true             2h53m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             5h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             5h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             5h53m
   compute-2.example.com         provisioned              compute-2-l2zmb            true             5h53m

7. After the RHCOS installation, verify that the BareMetalHost is added to the cluster:

   $ oc get nodes

   Example output

   NAME                           STATUS      ROLES     AGE   VERSION
   control-plane-1.example.com    available   master    4m2s  v1.29.4
   control-plane-2.example.com    available   master    141m  v1.29.4
   control-plane-3.example.com    available   master    141m  v1.29.4
   compute-1.example.com          available   worker    87m   v1.29.4
   compute-2.example.com          available   worker    87m   v1.29.4

   Note

   After replacement of the new control plane node, the etcd pod running in the new node is in crashloopback status. See "Replacing an unhealthy etcd member" in the Additional resources section for more information.

Procedure

1. Edit the provisioning custom resource (CR) to enable deploying with Virtual Media on the baremetal network:

   oc edit provisioning

     apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       creationTimestamp: "2021-08-05T18:51:50Z"
       finalizers:
       - provisioning.metal3.io
       generation: 8
       name: provisioning-configuration
       resourceVersion: "551591"
       uid: f76e956f-24c6-4361-aa5b-feaf72c5b526
     spec:
       provisioningDHCPRange: 172.22.0.10,172.22.0.254
       provisioningIP: 172.22.0.3
       provisioningInterface: enp1s0
       provisioningNetwork: Managed
       provisioningNetworkCIDR: 172.22.0.0/24
       virtualMediaViaExternalNetwork: true 1
     status:
       generations:
       - group: apps
         hash: ""
         lastGeneration: 7
         name: metal3
         namespace: openshift-machine-api
         resource: deployments
       - group: apps
         hash: ""
         lastGeneration: 1
         name: metal3-image-cache
         namespace: openshift-machine-api
         resource: daemonsets
       observedGeneration: 8
       readyReplicas: 0

   1

   Add virtualMediaViaExternalNetwork: true to the provisioning CR.

2. If the image URL exists, edit the machineset to use the API VIP address. This step only applies to clusters installed in versions 4.9 or earlier.

   oc edit machineset

     apiVersion: machine.openshift.io/v1beta1
     kind: MachineSet
     metadata:
       creationTimestamp: "2021-08-05T18:51:52Z"
       generation: 11
       labels:
         machine.openshift.io/cluster-api-cluster: ostest-hwmdt
         machine.openshift.io/cluster-api-machine-role: worker
         machine.openshift.io/cluster-api-machine-type: worker
       name: ostest-hwmdt-worker-0
       namespace: openshift-machine-api
       resourceVersion: "551513"
       uid: fad1c6e0-b9da-4d4a-8d73-286f78788931
     spec:
       replicas: 2
       selector:
         matchLabels:
           machine.openshift.io/cluster-api-cluster: ostest-hwmdt
           machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
       template:
         metadata:
           labels:
             machine.openshift.io/cluster-api-cluster: ostest-hwmdt
             machine.openshift.io/cluster-api-machine-role: worker
             machine.openshift.io/cluster-api-machine-type: worker
             machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
         spec:
           metadata: {}
           providerSpec:
             value:
               apiVersion: baremetal.cluster.k8s.io/v1alpha1
               hostSelector: {}
               image:
                 checksum: http:/172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2.<md5sum> 1
                 url: http://172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2 2
               kind: BareMetalMachineProviderSpec
               metadata:
                 creationTimestamp: null
               userData:
                 name: worker-user-data
     status:
       availableReplicas: 2
       fullyLabeledReplicas: 2
       observedGeneration: 11
       readyReplicas: 2
       replicas: 2

   1

   Edit the checksum URL to use the API VIP address.

   2

   Edit the url URL to use the API VIP address.

1. Get the bare-metal hosts running in the openshift-machine-api namespace:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                 STATUS   PROVISIONING STATUS      CONSUMER
   openshift-master-0   OK       externally provisioned   openshift-zpwpq-master-0
   openshift-master-1   OK       externally provisioned   openshift-zpwpq-master-1
   openshift-master-2   OK       externally provisioned   openshift-zpwpq-master-2
   openshift-worker-0   OK       provisioned              openshift-zpwpq-worker-0-lv84n
   openshift-worker-1   OK       provisioned              openshift-zpwpq-worker-0-zd8lm
   openshift-worker-2   error    registering

2. To see more detailed information about the status of the failing host, run the following command replacing <bare_metal_host_name> with the name of the host:

   $ oc get -n openshift-machine-api bmh <bare_metal_host_name> -o yaml

   Example output

   ...
   status:
     errorCount: 12
     errorMessage: MAC address b4:96:91:1d:7c:20 conflicts with existing node openshift-worker-1
     errorType: registration error
   ...

Procedure

1. Ensure the STATE is available before provisioning the bare metal node.

   $  oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME              STATE     ONLINE ERROR  AGE
   openshift-worker  available true          34h

2. Get a count of the number of worker nodes.

   $ oc get nodes

   NAME                                                STATUS   ROLES           AGE     VERSION
   openshift-master-1.openshift.example.com            Ready    master          30h     v1.29.4
   openshift-master-2.openshift.example.com            Ready    master          30h     v1.29.4
   openshift-master-3.openshift.example.com            Ready    master          30h     v1.29.4
   openshift-worker-0.openshift.example.com            Ready    worker          30h     v1.29.4
   openshift-worker-1.openshift.example.com            Ready    worker          30h     v1.29.4

3. Get the compute machine set.

   $ oc get machinesets -n openshift-machine-api

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   ...
   openshift-worker-0.example.com      1         1         1       1           55m
   openshift-worker-1.example.com      1         1         1       1           55m

4. Increase the number of worker nodes by one.

   $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

   Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the compute machine set from the previous step.

5. Check the status of the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number. The STATE changes from ready to provisioning.

   NAME                    STATE             CONSUMER                          ONLINE   ERROR
   openshift-worker-<num>  provisioning      openshift-worker-<num>-65tjz      true

   The provisioning status remains until the OpenShift Container Platform cluster provisions the node. This can take 30 minutes or more. After the node is provisioned, the state will change to provisioned.

   NAME                    STATE             CONSUMER                          ONLINE   ERROR
   openshift-worker-<num>  provisioned       openshift-worker-<num>-65tjz      true

6. After provisioning completes, ensure the bare metal node is ready.

   $ oc get nodes

   NAME                                          STATUS   ROLES   AGE     VERSION
   openshift-master-1.openshift.example.com      Ready    master  30h     v1.29.4
   openshift-master-2.openshift.example.com      Ready    master  30h     v1.29.4
   openshift-master-3.openshift.example.com      Ready    master  30h     v1.29.4
   openshift-worker-0.openshift.example.com      Ready    worker  30h     v1.29.4
   openshift-worker-1.openshift.example.com      Ready    worker  30h     v1.29.4
   openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.29.4

   You can also check the kubelet.

   $ ssh openshift-worker-<num>

   [kni@openshift-worker-<num>]$ journalctl -fu kubelet