Chapter 8. Deploying installer-provisioned clusters on bare metal

Procedure

1. Boot the node and enter the BIOS menu.
2. Set the node’s boot mode to UEFI Enabled.
3. Enable Secure Boot.

Procedure

1. Boot the node and select the BIOS menu.
2. Under the Advanced tab, select CSM Configuration from the list.

3. Enable the Launch CSM option and set the following values:

   Item	Value
   Boot option filter	UEFI and Legacy
   Launch PXE OpROM Policy	UEFI only
   Launch Storage OpROM policy	UEFI only
   Other PCI device ROM priority	UEFI only

Procedure

1. Log in to the provisioner node via ssh.

2. Create a non-root user (kni) and provide that user with sudo privileges:

   # useradd kni
   # passwd kni
   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni
   # chmod 0440 /etc/sudoers.d/kni

3. Create an ssh key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

4. Log in as the new user on the provisioner node:

   # su - kni
   $

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach
   $ sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhel-8-for-x86_64-baseos-rpms

   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

7. Modify the user to add the libvirt group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

8. Restart firewalld and enable the http service:

   $ sudo systemctl start firewalld
   $ sudo firewall-cmd --zone=public --add-service=http --permanent
   $ sudo firewall-cmd --reload

9. Start and enable the libvirtd service:

   $ sudo systemctl enable libvirtd --now

10. Create the default storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images
    $ sudo virsh pool-start default
    $ sudo virsh pool-autostart default

11. Configure networking.

    Note

    You can also configure networking from the web console.

    Export the baremetal network NIC name:

    $ export PUB_CONN=<baremetal_nic_name>

    Configure the baremetal network:

    $ sudo nohup bash -c "
        nmcli con down \"$PUB_CONN\"
        nmcli con delete \"$PUB_CONN\"
        # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
        nmcli con down \"System $PUB_CONN\"
        nmcli con delete \"System $PUB_CONN\"
        nmcli connection add ifname baremetal type bridge con-name baremetal
        nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
        pkill dhclient;dhclient baremetal
    "

    If you are deploying with a provisioning network, export the provisioning network NIC name:

    $ export PROV_CONN=<prov_nic_name>

    If you are deploying with a provisioning network, configure the provisioning network:

    $ sudo nohup bash -c "
        nmcli con down \"$PROV_CONN\"
        nmcli con delete \"$PROV_CONN\"
        nmcli connection add ifname provisioning type bridge con-name provisioning
        nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
        nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
        nmcli con down provisioning
        nmcli con up provisioning
    "

    Note

    The ssh connection might disconnect after executing these steps.

    The IPv6 address can be any address as long as it is not routable via the baremetal network.

    Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

12. Configure the IPv4 address on the provisioning network connection.

    $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

13. ssh back into the provisioner node (if required).

    # ssh kni@provisioner.<cluster-name>.<domain>

14. Verify the connection bridges have been properly created.

    $ sudo nmcli con show

    NAME               UUID                                  TYPE      DEVICE
    baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
    provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
    virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
    bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
    bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

15. Create a pull-secret.txt file.

    $ vim pull-secret.txt

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure, and scroll down to the Downloads section. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install
   $ export pullsecret_file=~/pull-secret.txt
   $ export extract_dir=$(pwd)

2. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

3. Extract the installer:

   $ sudo cp oc /usr/local/bin
   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
   $ sudo cp openshift-baremetal-install /usr/local/bin

Procedure

1. Install podman:

   $ sudo dnf install -y podman

2. Open firewall port 8080 to be used for RHCOS image caching:

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   $ sudo firewall-cmd --reload

3. Create a directory to store the bootstraposimage and clusterosimage:

   $ mkdir /home/kni/rhcos_image_cache

4. Set the appropriate SELinux context for the newly created directory:

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"

   $ sudo restorecon -Rv rhcos_image_cache/

5. Get the commit ID from the installer:

   $ export COMMIT_ID=$(/usr/local/bin/openshift-baremetal-install version | grep '^built from commit' | awk '{print $4}')

   The ID determines which images the installer needs to download.

6. Get the URI for the RHCOS image that the installer will deploy on the nodes:

   $ export RHCOS_OPENSTACK_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.openstack.path | sed 's/"//g')

7. Get the URI for the RHCOS image that the installer will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.qemu.path | sed 's/"//g')

8. Get the path where the images are published:

   $ export RHCOS_PATH=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json | jq .baseURI | sed 's/"//g')

9. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM:

   $ export RHCOS_QEMU_SHA_UNCOMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.qemu["uncompressed-sha256"]')

10. Get the SHA hash for the RHCOS image that will be deployed on the nodes:

    $ export RHCOS_OPENSTACK_SHA_COMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.openstack.sha256')

11. Download the images and place them in the /home/kni/rhcos_image_cache directory:

    $ curl -L ${RHCOS_PATH}${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_URI}

    $ curl -L ${RHCOS_PATH}${RHCOS_OPENSTACK_URI} -o /home/kni/rhcos_image_cache/${RHCOS_OPENSTACK_URI}

12. Confirm SELinux type is of httpd_sys_content_t for the newly created files:

    $ ls -Z /home/kni/rhcos_image_cache

13. Create the pod:

    $ podman run -d --name rhcos_image_cache \
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    quay.io/centos7/httpd-24-centos7:latest

    The above command creates a caching webserver with the name rhcos_image_cache, which serves the images for deployment. The first image ${RHCOS_PATH}${RHCOS_QEMU_URI}?sha256=${RHCOS_QEMU_SHA_UNCOMPRESSED} is the bootstrapOSImage and the second image ${RHCOS_PATH}${RHCOS_OPENSTACK_URI}?sha256=${RHCOS_OPENSTACK_SHA_COMPRESSED} is the clusterOSImage in the install-config.yaml file.

14. Generate the bootstrapOSImage and clusterOSImage configuration:

    $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)

    $ export RHCOS_OPENSTACK_SHA256=$(zcat /home/kni/rhcos_image_cache/${RHCOS_OPENSTACK_URI} | sha256sum | awk '{print $1}')

    $ export RHCOS_QEMU_SHA256=$(zcat /home/kni/rhcos_image_cache/${RHCOS_QEMU_URI} | sha256sum | awk '{print $1}')

    $ export CLUSTER_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_OPENSTACK_URI}?sha256=${RHCOS_OPENSTACK_SHA256}"

    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_URI}?sha256=${RHCOS_QEMU_SHA256}"

    $ echo "${RHCOS_OPENSTACK_SHA256}  ${RHCOS_OPENSTACK_URI}" > /home/kni/rhcos_image_cache/rhcos-ootpa-latest.qcow2.sha256sum

    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"

    $ echo "    clusterOSImage=${CLUSTER_OS_IMAGE}"

15. Add the required configuration to the install-config.yaml file under platform.baremetal:

    platform:
      baremetal:
        bootstrapOSImage: http://<BAREMETAL_IP>:8080/<RHCOS_QEMU_URI>?sha256=<RHCOS_QEMU_SHA256>
        clusterOSImage: http://<BAREMETAL_IP>:8080/<RHCOS_OPENSTACK_URI>?sha256=<RHCOS_OPENSTACK_SHA256>

    See the "Configuration files" section for additional details.

Procedure

1. Create a router-replicas.yaml file.

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: <num-of-router-pods>
     endpointPublishingStrategy:
       type: HostNetwork
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""

   Note

   Replace <num-of-router-pods> with an appropriate value. If working with just one worker node, set replicas: to 1. If working with more than 3 worker nodes, you can increase replicas: from the default value 2 as appropriate.

2. Save and copy the router-replicas.yaml file to the clusterconfigs/openshift directory.

   cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml

Procedure

1. Check the network interface configuration on the node.
2. Turn off the DHCP server and reboot the OpenShift Container Platform node and ensure that the network configuration works properly.

Procedure

1. Remove or reformat the disks for the bootstrap, control plane (also known as master) node, and worker nodes. If you are working in a hypervisor environment, you must add any disks you removed.

2. Delete the artifacts that the previous installation generated:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

3. Generate new manifests and Ignition config files. See “Creating the Kubernetes manifest and Ignition config files" for more information.
4. Upload the new bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. This will overwrite the previous Ignition files.

Procedure

1. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all worker nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

3. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the worker nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

4. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

5. Apply the 99-master-chrony-conf-override.yaml policy to the control plane nodes.

   $ oc apply -f 99-master-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-master-chrony-conf-override created

6. Apply the 99-worker-chrony-conf-override.yaml policy to the worker nodes.

   $ oc apply -f 99-worker-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-worker-chrony-conf-override created

7. Check the status of the applied NTP settings.

   $ oc describe machineconfigpool

Procedure

1. When setting the provisioningInterface setting, first identify the provisioning interface name for the cluster nodes. For example, eth0 or eno1.
2. Enable the Preboot eXecution Environment (PXE) on the provisioning network interface of the cluster nodes.

3. Retrieve the current state of the provisioning network and save it to a provisioning custom resource (CR) file:

   $ oc get provisioning -o yaml > enable-provisioning-nw.yaml

4. Modify the provisioning CR file:

   $ vim ~/enable-provisioning-nw.yaml

   Scroll down to the provisioningNetwork configuration setting and change it from Disabled to Managed. Then, add the provisioningOSDownloadURL, provisioningIP, provisioningNetworkCIDR, provisioningDHCPRange, provisioningInterface, and watchAllNameSpaces configuration settings after the provisioningNetwork setting. Provide appropriate values for each setting.

   apiVersion: v1
   items:
   - apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       name: provisioning-configuration
     spec:
       provisioningNetwork: 1
       provisioningOSDownloadURL: 2
       provisioningIP: 3
       provisioningNetworkCIDR: 4
       provisioningDHCPRange: 5
       provisioningInterface: 6
       watchAllNameSpaces: 7

   1

   The provisioningNetwork is one of Managed, Unmanaged, or Disabled. When set to Managed, Metal3 manages the provisioning network and the CBO deploys the Metal3 pod with a configured DHCP server. When set to Unmanaged, the system administrator configures the DHCP server manually.

   2

   The provisioningOSDownloadURL is a valid HTTPS URL with a valid sha256 checksum that enables the Metal3 pod to download a qcow2 operating system image ending in .qcow2.gz or .qcow2.xz. This field is required whether the provisioning network is Managed, Unmanaged, or Disabled. For example: http://192.168.0.1/images/rhcos-<version>.x86_64.qcow2.gz?sha256=<sha>.

   3

   The provisioningIP is the static IP address that the DHCP server and ironic use to provision the network. This static IP address must be within the provisioning subnet, and outside of the DHCP range. If you configure this setting, it must have a valid IP address even if the provisioning network is Disabled. The static IP address is bound to the metal3 pod. If the metal3 pod fails and moves to another server, the static IP address also moves to the new server.

   4

   The Classless Inter-Domain Routing (CIDR) address. If you configure this setting, it must have a valid CIDR address even if the provisioning network is Disabled. For example: 192.168.0.1/24.

   5

   The DHCP range. This setting is only applicable to a Managed provisioning network. Omit this configuration setting if the provisioning network is Disabled. For example: 192.168.0.64, 192.168.0.253.

   6

   The NIC name for the provisioning interface on cluster nodes. The provisioningInterface setting is only applicable to Managed and Unmanaged provisioning networks. Omit the provisioningInterface configuration setting if the provisioning network is Disabled. Omit the provisioningInterface configuration setting to use the bootMACAddress configuration setting instead.

   7

   Set this setting to true if you want metal3 to watch namespaces other than the default openshift-machine-api namespace. The default value is false.

5. Save the changes to the provisioning CR file.

6. Apply the provisioning CR file to the cluster:

   $ oc apply -f enable-provisioning-nw.yaml

Procedure

1. Enable access to the cluster from your load balancer on ports 6443, 443, and 80.

   As an example, note this HAProxy configuration:

   A section of a sample HAProxy configuration

   ...
   listen my-cluster-api-6443
       bind 0.0.0.0:6443
       mode tcp
       balance roundrobin
       server my-cluster-master-2 192.0.2.2:6443 check
       server my-cluster-master-0 192.0.2.3:6443 check
       server my-cluster-master-1 192.0.2.1:6443 check
   listen my-cluster-apps-443
           bind 0.0.0.0:443
           mode tcp
           balance roundrobin
           server my-cluster-worker-0 192.0.2.6:443 check
           server my-cluster-worker-1 192.0.2.5:443 check
           server my-cluster-worker-2 192.0.2.4:443 check
   listen my-cluster-apps-80
           bind 0.0.0.0:80
           mode tcp
           balance roundrobin
           server my-cluster-worker-0 192.0.2.7:80 check
           server my-cluster-worker-1 192.0.2.9:80 check
           server my-cluster-worker-2 192.0.2.8:80 check

2. Add records to your DNS server for the cluster API and apps over the load balancer. For example:

   <load_balancer_ip_address> api.<cluster_name>.<base_domain>
   <load_balancer_ip_address> apps.<cluster_name>.<base_domain>

3. From a command line, use curl to verify that the external load balancer and DNS configuration are operational.

   1. Verify that the cluster API is accessible:

      $ curl https://<loadbalancer_ip_address>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }

   2. Verify that cluster applications are accessible:

      Note

      You can also verify application accessibility by opening the OpenShift Container Platform console in a web browser.

      $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, you receive an HTTP response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

Procedure

1. Get the oc binary, if needed. It should already exist on the provisioner node.

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc

   $ sudo cp oc /usr/local/bin

2. Power off the bare metal node by using the baseboard management controller, and ensure it is off.

3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the user name and password:

   $ echo -ne "root" | base64

   $ echo -ne "password" | base64

4. Create a configuration file for the bare metal node.

   $ vim bmh.yaml

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-worker-<num>-bmc-secret
   type: Opaque
   data:
     username: <base64-of-uid>
     password: <base64-of-pwd>
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: openshift-worker-<num>
   spec:
     online: true
     bootMACAddress: <NIC1-mac-address>
     bmc:
       address: <protocol>://<bmc-ip>
       credentialsName: openshift-worker-<num>-bmc-secret

   Replace <num> for the worker number of the bare metal node in the two name fields and the credentialsName field. Replace <base64-of-uid> with the base64 string of the user name. Replace <base64-of-pwd> with the base64 string of the password. Replace <NIC1-mac-address> with the MAC address of the bare metal node’s first NIC.

   See the BMC addressing section for additional BMC configuration options. Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc-ip> with the IP address of the bare metal node’s baseboard management controller.

   Note

   If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See Diagnosing a host duplicate MAC address for more information.

5. Create the bare metal node.

   $ oc -n openshift-machine-api create -f bmh.yaml

   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

   Where <num> will be the worker number.

6. Power up and inspect the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

Procedure

1. Ensure that the Bare Metal Operator is available:

   $ oc get clusteroperator baremetal

   Example output

   NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   baremetal   4.8.0     True        False         False      3d15h

2. Remove the old BareMetalHost and Machine objects:

   $ oc delete bmh -n openshift-machine-api <host_name>
   $ oc delete machine -n openshift-machine-api <machine_name>

   Replace <host_name> with the name of the host and <machine_name> with the name of the machine. The machine name appears under the CONSUMER field.

   After you remove the BareMetalHost and Machine objects, then the machine controller automatically deletes the Node object.

3. Create the new BareMetalHost object and the secret to store the BMC credentials:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: control-plane-<num>-bmc-secret 1
     namespace: openshift-machine-api
   data:
     username: <base64_of_uid> 2
     password: <base64_of_pwd> 3
   type: Opaque
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: control-plane-<num> 4
     namespace: openshift-machine-api
   spec:
     automatedCleaningMode: disabled
     bmc:
       address: <protocol>://<bmc_ip> 5
       credentialsName: control-plane-<num>-bmc-secret 6
     bootMACAddress: <NIC1_mac_address> 7
     bootMode: UEFI
     externallyProvisioned: false
     hardwareProfile: unknown
     online: true
   EOF

   1 4 6

   Replace <num> for the control plane number of the bare metal node in the name fields and the credentialsName field.

   2

   Replace <base64_of_uid> with the base64 string of the user name.

   3

   Replace <base64_of_pwd> with the base64 string of the password.

   5

   Replace <protocol> with the BMC protocol, such as redfish, redfish-virtualmedia, idrac-virtualmedia, or others. Replace <bmc_ip> with the IP address of the bare metal node’s baseboard management controller. For additional BMC configuration options, see "BMC addressing" in the Additional resources section.

   7

   Replace <NIC1_mac_address> with the MAC address of the bare metal node’s first NIC.

   After the inspection is complete, the BareMetalHost object is created and available to be provisioned.

4. View available BareMetalHost objects:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   available                control-plane-1            true             1h10m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             4h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             4h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             4h53m
   compute-1.example.com         provisioned              compute-2-l2zmb            true             4h53m

   There are no MachineSet objects for control plane nodes, so you must create a Machine object instead. You can copy the providerSpec from another control plane Machine object.

5. Create a Machine object:

   $ cat <<EOF | oc apply -f -
   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     annotations:
       metal3.io/BareMetalHost: openshift-machine-api/control-plane-<num> 1
     labels:
       machine.openshift.io/cluster-api-cluster: control-plane-<num> 2
       machine.openshift.io/cluster-api-machine-role: master
       machine.openshift.io/cluster-api-machine-type: master
     name: control-plane-<num> 3
     namespace: openshift-machine-api
   spec:
     metadata: {}
     providerSpec:
       value:
         apiVersion: baremetal.cluster.k8s.io/v1alpha1
         customDeploy:
           method: install_coreos
         hostSelector: {}
         image:
           checksum: ""
           url: ""
         kind: BareMetalMachineProviderSpec
         metadata:
           creationTimestamp: null
         userData:
           name: master-user-data-managed
   EOF

   1 2 3

   Replace <num> for the control plane number of the bare metal node in the name, labels and annotations fields.

6. To view the BareMetalHost objects, run the following command:

   $ oc get bmh -A

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   provisioned              control-plane-1            true             2h53m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             5h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             5h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             5h53m
   compute-2.example.com         provisioned              compute-2-l2zmb            true             5h53m

7. After the RHCOS installation, verify that the BareMetalHost is added to the cluster:

   $ oc get nodes

   Example output

   NAME                           STATUS      ROLES     AGE   VERSION
   control-plane-1.example.com    available   master    4m2s  v1.18.2
   control-plane-2.example.com    available   master    141m  v1.18.2
   control-plane-3.example.com    available   master    141m  v1.18.2
   compute-1.example.com          available   worker    87m   v1.18.2
   compute-2.example.com          available   worker    87m   v1.18.2

   Note

   After replacement of the new control plane node, the etcd pod running in the new node is in crashloopback status. See "Replacing an unhealthy etcd member" in the Additional resources section for more information.

1. Get the bare-metal hosts running in the openshift-machine-api namespace:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                 STATUS   PROVISIONING STATUS      CONSUMER
   openshift-master-0   OK       externally provisioned   openshift-zpwpq-master-0
   openshift-master-1   OK       externally provisioned   openshift-zpwpq-master-1
   openshift-master-2   OK       externally provisioned   openshift-zpwpq-master-2
   openshift-worker-0   OK       provisioned              openshift-zpwpq-worker-0-lv84n
   openshift-worker-1   OK       provisioned              openshift-zpwpq-worker-0-zd8lm
   openshift-worker-2   error    registering

2. To see more detailed information about the status of the failing host, run the following command replacing <bare_metal_host_name> with the name of the host:

   $ oc get -n openshift-machine-api bmh <bare_metal_host_name> -o yaml

   Example output

   ...
   status:
     errorCount: 12
     errorMessage: MAC address b4:96:91:1d:7c:20 conflicts with existing node openshift-worker-1
     errorType: registration error
   ...

Procedure

1. Ensure the PROVISIONING STATUS is ready before provisioning the bare metal node.

   $  oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

2. Get a count of the number of worker nodes.

   $ oc get nodes

   NAME                                                STATUS   ROLES           AGE     VERSION
   provisioner.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-1.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-2.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-3.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-worker-0.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-worker-1.openshift.example.com            Ready    master          30h     v1.16.2

3. Get the machine set.

   $ oc get machinesets -n openshift-machine-api

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   ...
   openshift-worker-0.example.com      1         1         1       1           55m
   openshift-worker-1.example.com      1         1         1       1           55m

4. Increase the number of worker nodes by one.

   $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

   Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the machine set from the previous step.

5. Check the status of the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number. The status changes from ready to provisioning.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       provisioning          openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

   The provisioning status remains until the OpenShift Container Platform cluster provisions the node. This can take 30 minutes or more. After the node is provisioned, the status will change to provisioned.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       provisioned           openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

6. After provisioning completes, ensure the bare metal node is ready.

   $ oc get nodes

   NAME                                          STATUS   ROLES   AGE     VERSION
   provisioner.openshift.example.com             Ready    master  30h     v1.16.2
   openshift-master-1.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-master-2.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-master-3.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-0.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-1.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.16.2

   You can also check the kubelet.

   $ ssh openshift-worker-<num>

   [kni@openshift-worker-<num>]$ journalctl -fu kubelet

Procedure

1. Use the guidelines in YAML-tips.
2. Verify the YAML syntax is correct using syntax-check.

3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the install-config.yaml. For example:

   $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.x86_64.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7

   If the output is 200, there is a valid response from the webserver storing the bootstrap VM image.

Procedure

1. About 10 to 15 minutes after triggering the installation program, check to ensure the bootstrap VM is operational using the virsh command:

   $ sudo virsh list

    Id    Name                           State
    --------------------------------------------
    12    openshift-xf6fq-bootstrap      running

   Note

   The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap."

   If the bootstrap VM is not running after 10-15 minutes, troubleshoot why it is not running. Possible issues include:

2. Verify libvirtd is running on the system:

   $ systemctl status libvirtd

   ● libvirtd.service - Virtualization daemon
      Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
        Docs: man:libvirtd(8)
              https://libvirt.org
    Main PID: 9850 (libvirtd)
       Tasks: 20 (limit: 32768)
      Memory: 74.8M
      CGroup: /system.slice/libvirtd.service
              ├─ 9850 /usr/sbin/libvirtd

   If the bootstrap VM is operational, log in to it.

3. Use the virsh console command to find the IP address of the bootstrap VM:

   $ sudo virsh console example.com

   Connected to domain example.com
   Escape character is ^]
   Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
   SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
   SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
   SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
   ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
   ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
   localhost login:

   Important

   When deploying a OpenShift Container Platform cluster without the provisioning network, you must use a public IP address and not a private IP address like 172.22.0.2.

4. After you obtain the IP address, log in to the bootstrap VM using the ssh command:

   Note

   In the console output of the previous step, you can use the IPv6 IP address provided by ens3 or the IPv4 IP provided by ens4.

   $ ssh core@172.22.0.2

Procedure

1. Check the network connectivity to the provisioning network.
2. Ensure PXE is enabled on the NIC for the provisioning network and PXE is disabled for all other NICs.

3. Verify that the install-config.yaml configuration file has the proper hardware profile and boot MAC address for the NIC connected to the provisioning network. For example:

   control plane node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
   hardwareProfile: default          #control plane node settings

   Worker node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
   hardwareProfile: unknown          #worker node settings

Procedure

1. Hostname Resolution: Check the cluster nodes to ensure they have a fully qualified domain name, and not just localhost.localdomain. For example:

   $ hostname

   If a hostname is not set, set the correct hostname. For example:

   $ hostnamectl set-hostname <hostname>

2. Incorrect Name Resolution: Ensure that each node has the correct name resolution in the DNS server using dig and nslookup. For example:

   $ dig api.<cluster-name>.example.com

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster-name>.example.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
   ;; QUESTION SECTION:
   ;api.<cluster-name>.example.com. IN A
   
   ;; ANSWER SECTION:
   api.<cluster-name>.example.com. 10800 IN	A 10.19.13.86
   
   ;; AUTHORITY SECTION:
   <cluster-name>.example.com. 10800 IN NS	<cluster-name>.example.com.
   
   ;; ADDITIONAL SECTION:
   <cluster-name>.example.com. 10800 IN A	10.19.14.247
   
   ;; Query time: 0 msec
   ;; SERVER: 10.19.14.247#53(10.19.14.247)
   ;; WHEN: Tue May 19 20:30:59 UTC 2020
   ;; MSG SIZE  rcvd: 140

   The output in the foregoing example indicates that the appropriate IP address for the api.<cluster-name>.example.com VIP is 10.19.13.86. This IP address should reside on the baremetal network.

Procedure

1. Power off all bare metal nodes prior to installing the OpenShift Container Platform cluster:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

2. Remove all old bootstrap resources if any are left over from a previous deployment attempt:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Remove the following from the clusterconfigs directory to prevent Terraform from failing:

   $ rm -rf ~/clusterconfigs/auth ~/clusterconfigs/terraform* ~/clusterconfigs/tls ~/clusterconfigs/metadata.json

Procedure

1. Check to ensure authentication is successful:

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.json
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Note

   Example output of the variables used to mirror the install images:

   UPSTREAM_REPO=${RELEASE_IMAGE}
   LOCAL_REG=<registry_FQDN>:<registry_port>
   LOCAL_REPO='ocp4/openshift4'

   The values of RELEASE_IMAGE and VERSION were set during the Retrieving OpenShift Installer step of the Setting up the environment for an OpenShift installation section.

2. After mirroring the registry, confirm that you can access it in your disconnected environment:

   $ curl -k -u <user>:<password> https://registry.example.com:<registry-port>/v2/_catalog
   {"repositories":["<Repo-Name>"]}

Procedure

1. When the OpenShift Container Platform cluster nodes are installed appropriately, the following Ready state is seen within the STATUS column:

   $ oc get nodes

   NAME                   STATUS   ROLES           AGE  VERSION
   master-0.example.com   Ready    master,worker   4h   v1.16.2
   master-1.example.com   Ready    master,worker   4h   v1.16.2
   master-2.example.com   Ready    master,worker   4h   v1.16.2

2. Confirm the installer deployed all pods successfully. The following command removes any pods that are still running or have completed as part of the output.

   $ oc get pods --all-namespaces | grep -iv running | grep -iv complete