Chapter 8. Deploying installer-provisioned clusters on bare metal

Procedure

1. Boot the node and enter the BIOS menu.
2. Set the node’s boot mode to UEFI Enabled.
3. Enable Secure Boot.

Procedure

1. Boot the node and select the BIOS menu.
2. Under the Advanced tab, select CSM Configuration from the list.

3. Enable the Launch CSM option and set the following values:

   Expand

   Item	Value
   Boot option filter	UEFI and Legacy
   Launch PXE OpROM Policy	UEFI only
   Launch Storage OpROM policy	UEFI only
   Other PCI device ROM priority	UEFI only

   Show more

Procedure

1. Log in to the provisioner node via

   ssh

   .

2. Create a non-root user (

   kni

   ) and provide that user with

   sudo

   privileges:

   # useradd kni
   # passwd kni
   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni
   # chmod 0440 /etc/sudoers.d/kni

3. Create an

   ssh

   key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

4. Log in as the new user on the provisioner node:

   # su - kni
   $

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach
   $ sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhel-8-for-x86_64-baseos-rpms

   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

7. Modify the user to add the

   libvirt

   group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

8. Restart

   firewalld

   and enable the

   http

   service:

   $ sudo systemctl start firewalld
   $ sudo firewall-cmd --zone=public --add-service=http --permanent
   $ sudo firewall-cmd --reload

9. Start and enable the

   libvirtd

   service:

   $ sudo systemctl enable libvirtd --now

10. Create the

    default

    storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images
    $ sudo virsh pool-start default
    $ sudo virsh pool-autostart default

11. Configure networking.

    Note

    You can also configure networking from the web console.

    Export the

    baremetal

    network NIC name:

    $ export PUB_CONN=<baremetal_nic_name>

    Configure the

    baremetal

    network:

    $ sudo nohup bash -c "
        nmcli con down \"$PUB_CONN\"
        nmcli con delete \"$PUB_CONN\"
        # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
        nmcli con down \"System $PUB_CONN\"
        nmcli con delete \"System $PUB_CONN\"
        nmcli connection add ifname baremetal type bridge con-name baremetal
        nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
        pkill dhclient;dhclient baremetal
    "

    If you are deploying with a

    provisioning

    network, export the

    provisioning

    network NIC name:

    $ export PROV_CONN=<prov_nic_name>

    If you are deploying with a

    provisioning

    network, configure the

    provisioning

    network:

    $ sudo nohup bash -c "
        nmcli con down \"$PROV_CONN\"
        nmcli con delete \"$PROV_CONN\"
        nmcli connection add ifname provisioning type bridge con-name provisioning
        nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
        nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
        nmcli con down provisioning
        nmcli con up provisioning
    "

    Note

    The

    ssh

    connection might disconnect after executing these steps.

    The IPv6 address can be any address as long as it is not routable via the

    baremetal

    network.

    Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

12. Configure the IPv4 address on the

    provisioning

    network connection.

    $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

13. ssh

    back into the

    provisioner

    node (if required).

    # ssh kni@provisioner.<cluster-name>.<domain>

14. Verify the connection bridges have been properly created.

    $ sudo nmcli con show

    NAME               UUID                                  TYPE      DEVICE
    baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
    provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
    virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
    bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
    bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

15. Create a

    pull-secret.txt

    file.

    $ vim pull-secret.txt

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure, and scroll down to the Downloads section. Click Copy pull secret. Paste the contents into the

    pull-secret.txt

    file and save the contents in the

    kni

    user’s home directory.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install
   $ export pullsecret_file=~/pull-secret.txt
   $ export extract_dir=$(pwd)

2. Get the

   oc

   binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

3. Extract the installer:

   $ sudo cp oc /usr/local/bin
   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
   $ sudo cp openshift-baremetal-install /usr/local/bin

Procedure

1. Install

   podman

   :

   $ sudo dnf install -y podman

2. Open firewall port

   8080

   to be used for RHCOS image caching:

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   $ sudo firewall-cmd --reload

3. Create a directory to store the

   bootstraposimage

   and

   clusterosimage

   :

   $ mkdir /home/kni/rhcos_image_cache

4. Set the appropriate SELinux context for the newly created directory:

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"

   $ sudo restorecon -Rv rhcos_image_cache/

5. Get the commit ID from the installer:

   $ export COMMIT_ID=$(/usr/local/bin/openshift-baremetal-install version | grep '^built from commit' | awk '{print $4}')

   The ID determines which images the installer needs to download.

6. Get the URI for the RHCOS image that the installer will deploy on the nodes:

   $ export RHCOS_OPENSTACK_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.openstack.path | sed 's/"//g')

7. Get the URI for the RHCOS image that the installer will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.qemu.path | sed 's/"//g')

8. Get the path where the images are published:

   $ export RHCOS_PATH=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json | jq .baseURI | sed 's/"//g')

9. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM:

   $ export RHCOS_QEMU_SHA_UNCOMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.qemu["uncompressed-sha256"]')

10. Get the SHA hash for the RHCOS image that will be deployed on the nodes:

    $ export RHCOS_OPENSTACK_SHA_COMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.openstack.sha256')

11. Download the images and place them in the

    /home/kni/rhcos_image_cache

    directory:

    $ curl -L ${RHCOS_PATH}${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_URI}

    $ curl -L ${RHCOS_PATH}${RHCOS_OPENSTACK_URI} -o /home/kni/rhcos_image_cache/${RHCOS_OPENSTACK_URI}

12. Confirm SELinux type is of

    httpd_sys_content_t

    for the newly created files:

    $ ls -Z /home/kni/rhcos_image_cache

13. Create the pod:

    $ podman run -d --name rhcos_image_cache \
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    quay.io/centos7/httpd-24-centos7:latest

    The above command creates a caching webserver with the name

    rhcos_image_cache

    , which serves the images for deployment. The first image

    ${RHCOS_PATH}${RHCOS_QEMU_URI}?sha256=${RHCOS_QEMU_SHA_UNCOMPRESSED}

    is the

    bootstrapOSImage

    and the second image

    ${RHCOS_PATH}${RHCOS_OPENSTACK_URI}?sha256=${RHCOS_OPENSTACK_SHA_COMPRESSED}

    is the

    clusterOSImage

    in the

    install-config.yaml

    file.

14. Generate the

    bootstrapOSImage

    and

    clusterOSImage

    configuration:

    $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)

    $ export RHCOS_OPENSTACK_SHA256=$(zcat /home/kni/rhcos_image_cache/${RHCOS_OPENSTACK_URI} | sha256sum | awk '{print $1}')

    $ export RHCOS_QEMU_SHA256=$(zcat /home/kni/rhcos_image_cache/${RHCOS_QEMU_URI} | sha256sum | awk '{print $1}')

    $ export CLUSTER_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_OPENSTACK_URI}?sha256=${RHCOS_OPENSTACK_SHA256}"

    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_URI}?sha256=${RHCOS_QEMU_SHA256}"

    $ echo "${RHCOS_OPENSTACK_SHA256}  ${RHCOS_OPENSTACK_URI}" > /home/kni/rhcos_image_cache/rhcos-ootpa-latest.qcow2.sha256sum

    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"

    $ echo "    clusterOSImage=${CLUSTER_OS_IMAGE}"

15. Add the required configuration to the

    install-config.yaml

    file under

    platform.baremetal

    :

    platform:
      baremetal:
        bootstrapOSImage: http://<BAREMETAL_IP>:8080/<RHCOS_QEMU_URI>?sha256=<RHCOS_QEMU_SHA256>
        clusterOSImage: http://<BAREMETAL_IP>:8080/<RHCOS_OPENSTACK_URI>?sha256=<RHCOS_OPENSTACK_SHA256>

    See the "Configuration files" section for additional details.

Procedure

1. Create a

   router-replicas.yaml

   file.

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: <num-of-router-pods>
     endpointPublishingStrategy:
       type: HostNetwork
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""

   Note

   Replace

   <num-of-router-pods>

   with an appropriate value. If working with just one worker node, set

   replicas:

   to

   1

   . If working with more than 3 worker nodes, you can increase

   replicas:

   from the default value

   2

   as appropriate.

2. Save and copy the

   router-replicas.yaml

   file to the

   clusterconfigs/openshift

   directory.

   cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml

Procedure

1. Check the network interface configuration on the node.
2. Turn off the DHCP server and reboot the OpenShift Container Platform node and ensure that the network configuration works properly.

Procedure

1. Remove or reformat the disks for the bootstrap, control plane (also known as master) node, and worker nodes. If you are working in a hypervisor environment, you must add any disks you removed.

2. Delete the artifacts that the previous installation generated:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

3. Generate new manifests and Ignition config files. See “Creating the Kubernetes manifest and Ignition config files" for more information.
4. Upload the new bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. This will overwrite the previous Ignition files.

Procedure

1. Create a Butane config,

   99-master-chrony-conf-override.bu

   , including the contents of the

   chrony.conf

   file for the control plane nodes.
   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 

   1

   
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all worker nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

2. Use Butane to generate a

   MachineConfig

   object file,

   99-master-chrony-conf-override.yaml

   , containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

3. Create a Butane config,

   99-worker-chrony-conf-override.bu

   , including the contents of the

   chrony.conf

   file for the worker nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 

   1

   
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

4. Use Butane to generate a

   MachineConfig

   object file,

   99-worker-chrony-conf-override.yaml

   , containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

5. Apply the

   99-master-chrony-conf-override.yaml

   policy to the control plane nodes.

   $ oc apply -f 99-master-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-master-chrony-conf-override created

6. Apply the

   99-worker-chrony-conf-override.yaml

   policy to the worker nodes.

   $ oc apply -f 99-worker-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-worker-chrony-conf-override created

7. Check the status of the applied NTP settings.

   $ oc describe machineconfigpool

Procedure

1. When setting the

   provisioningInterface

   setting, first identify the provisioning interface name for the cluster nodes. For example,

   eth0

   or

   eno1

   .
2. Enable the Preboot eXecution Environment (PXE) on the

   provisioning

   network interface of the cluster nodes.

3. Retrieve the current state of the

   provisioning

   network and save it to a provisioning custom resource (CR) file:

   $ oc get provisioning -o yaml > enable-provisioning-nw.yaml

4. Modify the provisioning CR file:

   $ vim ~/enable-provisioning-nw.yaml

   Scroll down to the

   provisioningNetwork

   configuration setting and change it from

   Disabled

   to

   Managed

   . Then, add the

   provisioningOSDownloadURL

   ,

   provisioningIP

   ,

   provisioningNetworkCIDR

   ,

   provisioningDHCPRange

   ,

   provisioningInterface

   , and

   watchAllNameSpaces

   configuration settings after the

   provisioningNetwork

   setting. Provide appropriate values for each setting.

   apiVersion: v1
   items:
   - apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       name: provisioning-configuration
     spec:
       provisioningNetwork: 

   1

   
       provisioningOSDownloadURL: 

   2

   
       provisioningIP: 

   3

   
       provisioningNetworkCIDR: 

   4

   
       provisioningDHCPRange: 

   5

   
       provisioningInterface: 

   6

   
       watchAllNameSpaces: 

   7

   1

   The provisioningNetwork is one of Managed, Unmanaged, or Disabled. When set to Managed, Metal3 manages the provisioning network and the CBO deploys the Metal3 pod with a configured DHCP server. When set to Unmanaged, the system administrator configures the DHCP server manually.

   2

   The provisioningOSDownloadURL is a valid HTTPS URL with a valid sha256 checksum that enables the Metal3 pod to download a qcow2 operating system image ending in .qcow2.gz or .qcow2.xz. This field is required whether the provisioning network is Managed, Unmanaged, or Disabled. For example: http://192.168.0.1/images/rhcos-<version>.x86_64.qcow2.gz?sha256=<sha>.

   3

   The provisioningIP is the static IP address that the DHCP server and ironic use to provision the network. This static IP address must be within the provisioning subnet, and outside of the DHCP range. If you configure this setting, it must have a valid IP address even if the provisioning network is Disabled. The static IP address is bound to the metal3 pod. If the metal3 pod fails and moves to another server, the static IP address also moves to the new server.

   4

   The Classless Inter-Domain Routing (CIDR) address. If you configure this setting, it must have a valid CIDR address even if the provisioning network is Disabled. For example: 192.168.0.1/24.

   5

   The DHCP range. This setting is only applicable to a Managed provisioning network. Omit this configuration setting if the provisioning network is Disabled. For example: 192.168.0.64, 192.168.0.253.

   6

   The NIC name for the provisioning interface on cluster nodes. The provisioningInterface setting is only applicable to Managed and Unmanaged provisioning networks. Omit the provisioningInterface configuration setting if the provisioning network is Disabled. Omit the provisioningInterface configuration setting to use the bootMACAddress configuration setting instead.

   7

   Set this setting to true if you want metal3 to watch namespaces other than the default openshift-machine-api namespace. The default value is false.

5. Save the changes to the provisioning CR file.

6. Apply the provisioning CR file to the cluster:

   $ oc apply -f enable-provisioning-nw.yaml

Procedure

1. Enable access to the cluster from your load balancer on ports 6443, 443, and 80.

   As an example, note this HAProxy configuration:

   A section of a sample HAProxy configuration

   ...
   listen my-cluster-api-6443
       bind 0.0.0.0:6443
       mode tcp
       balance roundrobin
       server my-cluster-master-2 192.0.2.2:6443 check
       server my-cluster-master-0 192.0.2.3:6443 check
       server my-cluster-master-1 192.0.2.1:6443 check
   listen my-cluster-apps-443
           bind 0.0.0.0:443
           mode tcp
           balance roundrobin
           server my-cluster-worker-0 192.0.2.6:443 check
           server my-cluster-worker-1 192.0.2.5:443 check
           server my-cluster-worker-2 192.0.2.4:443 check
   listen my-cluster-apps-80
           bind 0.0.0.0:80
           mode tcp
           balance roundrobin
           server my-cluster-worker-0 192.0.2.7:80 check
           server my-cluster-worker-1 192.0.2.9:80 check
           server my-cluster-worker-2 192.0.2.8:80 check

2. Add records to your DNS server for the cluster API and apps over the load balancer. For example:

   <load_balancer_ip_address> api.<cluster_name>.<base_domain>
   <load_balancer_ip_address> apps.<cluster_name>.<base_domain>

3. From a command line, use

   curl

   to verify that the external load balancer and DNS configuration are operational.

   1. Verify that the cluster API is accessible:

      $ curl https://<loadbalancer_ip_address>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }

   2. Verify that cluster applications are accessible:

      Note

      You can also verify application accessibility by opening the OpenShift Container Platform console in a web browser.

      $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, you receive an HTTP response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

Procedure

1. Get the

   oc

   binary, if needed. It should already exist on the provisioner node.

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc

   $ sudo cp oc /usr/local/bin

2. Power off the bare metal node by using the baseboard management controller, and ensure it is off.

3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create

   base64

   strings from the user name and password:

   $ echo -ne "root" | base64

   $ echo -ne "password" | base64

4. Create a configuration file for the bare metal node.

   $ vim bmh.yaml

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-worker-<num>-bmc-secret
   type: Opaque
   data:
     username: <base64-of-uid>
     password: <base64-of-pwd>
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: openshift-worker-<num>
   spec:
     online: true
     bootMACAddress: <NIC1-mac-address>
     bmc:
       address: <protocol>://<bmc-ip>
       credentialsName: openshift-worker-<num>-bmc-secret

   Replace

   <num>

   for the worker number of the bare metal node in the two

   name

   fields and the

   credentialsName

   field. Replace

   <base64-of-uid>

   with the

   base64

   string of the user name. Replace

   <base64-of-pwd>

   with the

   base64

   string of the password. Replace

   <NIC1-mac-address>

   with the MAC address of the bare metal node’s first NIC.

   See the BMC addressing section for additional BMC configuration options. Replace

   <protocol>

   with the BMC protocol, such as IPMI, RedFish, or others. Replace

   <bmc-ip>

   with the IP address of the bare metal node’s baseboard management controller.
   Note

   If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See Diagnosing a host duplicate MAC address for more information.

5. Create the bare metal node.

   $ oc -n openshift-machine-api create -f bmh.yaml

   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

   Where

   <num>

   will be the worker number.

6. Power up and inspect the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where

   <num>

   is the worker node number.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

Procedure

1. Ensure that the Bare Metal Operator is available:

   $ oc get clusteroperator baremetal

   Example output

   NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   baremetal   4.8.0     True        False         False      3d15h

2. Remove the old

   BareMetalHost

   and

   Machine

   objects:

   $ oc delete bmh -n openshift-machine-api <host_name>
   $ oc delete machine -n openshift-machine-api <machine_name>

   Replace

   <host_name>

   with the name of the host and

   <machine_name>

   with the name of the machine. The machine name appears under the

   CONSUMER

   field.

   After you remove the

   BareMetalHost

   and

   Machine

   objects, then the machine controller automatically deletes the

   Node

   object.

3. Create the new

   BareMetalHost

   object and the secret to store the BMC credentials:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: control-plane-<num>-bmc-secret 

   1

   
     namespace: openshift-machine-api
   data:
     username: <base64_of_uid> 

   2

   
     password: <base64_of_pwd> 

   3

   
   type: Opaque
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: control-plane-<num> 

   4

   
     namespace: openshift-machine-api
   spec:
     automatedCleaningMode: disabled
     bmc:
       address: <protocol>://<bmc_ip> 

   5

   
       credentialsName: control-plane-<num>-bmc-secret 

   6

   
     bootMACAddress: <NIC1_mac_address> 

   7

   
     bootMode: UEFI
     externallyProvisioned: false
     hardwareProfile: unknown
     online: true
   EOF

   1 4 6

   Replace <num> for the control plane number of the bare metal node in the name fields and the credentialsName field.

   2

   Replace <base64_of_uid> with the base64 string of the user name.

   3

   Replace <base64_of_pwd> with the base64 string of the password.

   5

   Replace <protocol> with the BMC protocol, such as redfish, redfish-virtualmedia, idrac-virtualmedia, or others. Replace <bmc_ip> with the IP address of the bare metal node’s baseboard management controller. For additional BMC configuration options, see "BMC addressing" in the Additional resources section.

   7

   Replace <NIC1_mac_address> with the MAC address of the bare metal node’s first NIC.

   After the inspection is complete, the

   BareMetalHost

   object is created and available to be provisioned.

4. View available

   BareMetalHost

   objects:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   available                control-plane-1            true             1h10m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             4h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             4h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             4h53m
   compute-1.example.com         provisioned              compute-2-l2zmb            true             4h53m

   There are no

   MachineSet

   objects for control plane nodes, so you must create a

   Machine

   object instead. You can copy the

   providerSpec

   from another control plane

   Machine

   object.

5. Create a

   Machine

   object:

   $ cat <<EOF | oc apply -f -
   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     annotations:
       metal3.io/BareMetalHost: openshift-machine-api/control-plane-<num> 

   1

   
     labels:
       machine.openshift.io/cluster-api-cluster: control-plane-<num> 

   2

   
       machine.openshift.io/cluster-api-machine-role: master
       machine.openshift.io/cluster-api-machine-type: master
     name: control-plane-<num> 

   3

   
     namespace: openshift-machine-api
   spec:
     metadata: {}
     providerSpec:
       value:
         apiVersion: baremetal.cluster.k8s.io/v1alpha1
         customDeploy:
           method: install_coreos
         hostSelector: {}
         image:
           checksum: ""
           url: ""
         kind: BareMetalMachineProviderSpec
         metadata:
           creationTimestamp: null
         userData:
           name: master-user-data-managed
   EOF

   1 2 3

   Replace <num> for the control plane number of the bare metal node in the name, labels and annotations fields.

6. To view the

   BareMetalHost

   objects, run the following command:

   $ oc get bmh -A

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   provisioned              control-plane-1            true             2h53m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             5h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             5h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             5h53m
   compute-2.example.com         provisioned              compute-2-l2zmb            true             5h53m

7. After the RHCOS installation, verify that the

   BareMetalHost

   is added to the cluster:

   $ oc get nodes

   Example output

   NAME                           STATUS      ROLES     AGE   VERSION
   control-plane-1.example.com    available   master    4m2s  v1.18.2
   control-plane-2.example.com    available   master    141m  v1.18.2
   control-plane-3.example.com    available   master    141m  v1.18.2
   compute-1.example.com          available   worker    87m   v1.18.2
   compute-2.example.com          available   worker    87m   v1.18.2

   Note

   After replacement of the new control plane node, the etcd pod running in the new node is in

   crashloopback

   status. See "Replacing an unhealthy etcd member" in the Additional resources section for more information.