Deploying installer-provisioned clusters on bare metal

Procedure

1. Log in to the provisioner node via ssh.

2. Create a non-root user (kni) and provide that user with sudo privileges:

   # useradd kni

   Copy to Clipboard Toggle word wrap

   # passwd kni

   Copy to Clipboard Toggle word wrap

   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni

   Copy to Clipboard Toggle word wrap

   # chmod 0440 /etc/sudoers.d/kni

   Copy to Clipboard Toggle word wrap

3. Create an ssh key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

   Copy to Clipboard Toggle word wrap

4. Log in as the new user on the provisioner node:

   # su - kni

   Copy to Clipboard Toggle word wrap

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach

   Copy to Clipboard Toggle word wrap

   $ sudo subscription-manager repos --enable=rhel-9-for-<architecture>-appstream-rpms --enable=rhel-9-for-<architecture>-baseos-rpms

   Copy to Clipboard Toggle word wrap
   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

   Copy to Clipboard Toggle word wrap

7. Modify the user to add the libvirt group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

   Copy to Clipboard Toggle word wrap

8. Restart firewalld and enable the http service:

   $ sudo systemctl start firewalld

   Copy to Clipboard Toggle word wrap

   $ sudo firewall-cmd --zone=public --add-service=http --permanent

   Copy to Clipboard Toggle word wrap

   $ sudo firewall-cmd --reload

   Copy to Clipboard Toggle word wrap

9. Start the modular libvirt daemon sockets:

   $ for drv in qemu interface network nodedev nwfilter secret storage; do sudo systemctl start virt${drv}d{,-ro,-admin}.socket; done

   Copy to Clipboard Toggle word wrap

10. Create the default storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images

    Copy to Clipboard Toggle word wrap

    $ sudo virsh pool-start default

    Copy to Clipboard Toggle word wrap

    $ sudo virsh pool-autostart default

    Copy to Clipboard Toggle word wrap

11. Create a pull-secret.txt file:

    $ vim pull-secret.txt

    Copy to Clipboard Toggle word wrap

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Procedure

1. Log in to the node by using the ssh command.

2. View the NTP servers available to the node by running the following command:

   $ chronyc sources

   Copy to Clipboard Toggle word wrap

   Example output

   MS Name/IP address         Stratum Poll Reach LastRx Last sample
   ===============================================================================
   ^+ time.cloudflare.com           3  10   377   187   -209us[ -209us] +/-   32ms
   ^+ t1.time.ir2.yahoo.com         2  10   377   185  -4382us[-4382us] +/-   23ms
   ^+ time.cloudflare.com           3  10   377   198   -996us[-1220us] +/-   33ms
   ^* brenbox.westnet.ie            1  10   377   193  -9538us[-9761us] +/-   24ms

   Copy to Clipboard Toggle word wrap

3. Use the ping command to ensure that the node can access an NTP server, for example:

   $ ping time.cloudflare.com

   Copy to Clipboard Toggle word wrap

   Example output

   PING time.cloudflare.com (162.159.200.123) 56(84) bytes of data.
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=1 ttl=54 time=32.3 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=2 ttl=54 time=30.9 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=3 ttl=54 time=36.7 ms
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Export the bare-metal network NIC name by running the following command:

   $ export PUB_CONN=<baremetal_nic_name>

   Copy to Clipboard Toggle word wrap

2. Configure the bare-metal network:

   Note

   The SSH connection might disconnect after executing these steps.

   1. For a network using DHCP, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge <con_name> baremetal bridge.stp no 

      1

      
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          pkill dhclient;dhclient baremetal
      "

      Copy to Clipboard Toggle word wrap

      1

      Replace <con_name> with the connection name.

   2. For a network using static IP addressing and no DHCP network, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge con-name baremetal bridge.stp no ipv4.method manual ipv4.addr "x.x.x.x/yy" ipv4.gateway "a.a.a.a" ipv4.dns "b.b.b.b" 

      1

      
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          nmcli con up baremetal
      "

      Copy to Clipboard Toggle word wrap

      1

      Replace <con_name> with the connection name. Replace x.x.x.x/yy with the IP address and CIDR for the network. Replace a.a.a.a with the network gateway. Replace b.b.b.b with the IP address of the DNS server.

3. Optional: If you are deploying with a provisioning network, export the provisioning network NIC name by running the following command:

   $ export PROV_CONN=<prov_nic_name>

   Copy to Clipboard Toggle word wrap

4. Optional: If you are deploying with a provisioning network, configure the provisioning network by running the following command:

   $ sudo nohup bash -c "
       nmcli con down \"$PROV_CONN\"
       nmcli con delete \"$PROV_CONN\"
       nmcli connection add ifname provisioning type bridge con-name provisioning
       nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
       nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
       nmcli con down provisioning
       nmcli con up provisioning
   "

   Copy to Clipboard Toggle word wrap
   Note

   The SSH connection might disconnect after executing these steps.

   The IPv6 address can be any address that is not routable through the bare-metal network.

   Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

5. Optional: If you are deploying with a provisioning network, configure the IPv4 address on the provisioning network connection by running the following command:

   $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

   Copy to Clipboard Toggle word wrap

6. SSH back into the provisioner node (if required) by running the following command:

   # ssh kni@provisioner.<cluster-name>.<domain>

   Copy to Clipboard Toggle word wrap

7. Verify that the connection bridges have been properly created by running the following command:

   $ sudo nmcli con show

   Copy to Clipboard Toggle word wrap

   Example output

   NAME               UUID                                  TYPE      DEVICE
   baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
   provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
   virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
   bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
   bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:
   - name: enp2s0 

   1

   
     type: ethernet 

   2

   
     state: up 

   3

   
     ipv4:
       enabled: false 

   4

   
     ipv6:
       enabled: false
   - name: br-ex
     type: ovs-bridge
     state: up
     ipv4:
       enabled: false
       dhcp: false
     ipv6:
       enabled: false
       dhcp: false
     bridge:
       options:
         mcast-snooping-enable: true
       port:
       - name: enp2s0 

   5

   
       - name: br-ex
   - name: br-ex
     type: ovs-interface
     state: up
     copy-mac-from: enp2s0
     ipv4:
       enabled: true
       dhcp: true
       auto-route-metric: 48 

   6

   
     ipv6:
       enabled: true
       dhcp: true
       auto-route-metric: 48
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Name of the interface.

   2

   The type of ethernet.

   3

   The requested state for the interface after creation.

   4

   Disables IPv4 and IPv6 in this example.

   5

   The node NIC to which the bridge attaches.

   6

   Set the parameter to 48 to ensure the br-ex default route always has the highest precedence (lowest metric). This configuration prevents routing conflicts with any other interfaces that are automatically configured by the NetworkManager service.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yaml | base64 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig manifest file and define a customized br-ex bridge network configuration analogous to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 10-br-ex-worker 

   1

   
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 

   2

   
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/worker-0.yml 

   3

   
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration>
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/worker-1.yml 

   4

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The name of the policy.

   2

   Writes the encoded base64 information to the specified path.

   3 4

   For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. The worker role is the default role for nodes in your cluster. The .yaml extension does not work when specifying the short hostname, hostname -s, path for each node or all nodes in the MachineConfig manifest file.

   If you have a single global configuration specified in an /etc/nmstate/openshift/cluster.yml configuration file that you want to apply to all nodes in your cluster, you do not need to specify the short hostname path for each node, such as /etc/nmstate/openshift/<node_hostname>.yml. For example:

   # ...
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration>
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. For each bare-metal host that exists in a cluster, in the install-config.yaml file for your cluster define a networkConfig section similar to the following example:

   apiVersion: v1
   kind: InstallConfig
   metadata:
     name: <cluster-name>
   # ...
   networkConfig:
     interfaces:
       - name: enp1s0 

   1

   
         type: ethernet
         state: up
         ipv4:
           dhcp: true
           enabled: true
         ipv6:
           enabled: false
       - name: enp2s0 

   2

   
         type: ethernet
         state: up
         mtu: 1500 

   3

   
         ipv4:
           dhcp: true
           enabled: true
         ipv6:
           dhcp: true
           enabled: true
       - name: enp3s0 

   4

   
         type: ethernet
         state: up
         mtu: 1500
         ipv4:
           enabled: false
         ipv6:
           enabled: false
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The interface for the provisioned network interface controller (NIC).

   2

   The first bonded interface that pulls in the Ignition config file for the bond interface.

   3

   Manually set the br-ex maximum transmission unit (MTU) on the bond ports.

   4

   The second bonded interface is part of a minimal configuration that pulls ignition during cluster installation.

2. Define each network interface in an NMState configuration file:

   Example NMState configuration file that defines many network interfaces

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
   # ...
   ovn:
     bridge-mappings:
       - localnet: localnet-network
         bridge: br-ex
         state: present
   interfaces:
     - name: br-ex
       type: ovs-bridge
       state: up
       bridge:
         allow-extra-patch-ports: true
         port:
           - name: br-ex
           - name: patch-ex-to-phy
       ovs-db:
         external_ids:
           bridge-uplink: "patch-ex-to-phy"
     - name: br-ex
       type: ovs-interface
       state: up
       mtu: 1500 

   1

   
       ipv4:
         enabled: true
         dhcp: true
         auto-route-metric: 48
       ipv6:
         enabled: false
         dhcp: false
         auto-route-metric: 48
     - name: br-phy
       type: ovs-bridge
       state: up
       bridge:
         allow-extra-patch-ports: true
         port:
           - name: patch-phy-to-ex
           - name: ovs-bond
             link-aggregation:
               mode: balance-slb
               port:
                 - name: enp2s0
                 - name: enp3s0
     - name: patch-ex-to-phy
       type: ovs-interface
       state: up
       patch:
         peer: patch-phy-to-ex
     - name: patch-phy-to-ex
       type: ovs-interface
       state: up
       patch:
         peer: patch-ex-to-phy
     - name: enp1s0
       type: ethernet
       state: up
       ipv4:
         dhcp: true
         enabled: true
       ipv6:
         enabled: false
     - name: enp2s0
       type: ethernet
       state: up
       mtu: 1500
       ipv4:
         enabled: false
       ipv6:
         enabled: false
     - name: enp3s0
       type: ethernet
       state: up
       mtu: 1500
       ipv4:
         enabled: false
       ipv6:
         enabled: false
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Manually set the br-ex MTU on the bond ports.

3. Use the base64 command to encode the interface content of the NMState configuration file:

   $ base64 -w0  <nmstate_configuration>.yml 

   1

   Copy to Clipboard Toggle word wrap

   1

   Where the -w0 option prevents line wrapping during the base64 encoding operation.

4. Create MachineConfig manifest files for the master role and the worker role. Ensure that you embed the base64-encoded string from an earlier command into each MachineConfig manifest file. The following example manifest file configures the master role for all nodes that exist in a cluster. You can also create a manifest file for master and worker roles specific to a node.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 10-br-ex-master 

   1

   
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 

   2

   
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml 

   3

   Copy to Clipboard Toggle word wrap

   1

   The name of the policy.

   2

   Writes the encoded base64 information to the specified path.

   3

   Specify the path to the cluster.yml file. For each node in your cluster, you can specify the short hostname path to your node, such as <node_short_hostname>.yml.

5. Save each MachineConfig manifest file to the ./<installation_directory>/manifests directory, where <installation_directory> is the directory in which the installation program creates files.

   The Machine Config Operator (MCO) takes the content from each manifest file and consistently applies the content to all selected nodes during a rolling update.

Procedure

1. Configure the first subnet to communicate with the second subnet:

   1. Log in as root to a control plane node by running the following command:

      $ sudo su -

      Copy to Clipboard Toggle word wrap

   2. Get the name of the network interface by running the following command:

      # nmcli dev status

      Copy to Clipboard Toggle word wrap

   3. Add a route to the second subnet (192.168.0.0) via the gateway by running the following command:

      # nmcli connection modify <interface_name> +ipv4.routes "192.168.0.0/24 via <gateway>"

      Copy to Clipboard Toggle word wrap

      Replace <interface_name> with the interface name. Replace <gateway> with the IP address of the actual gateway.

      Example

      # nmcli connection modify eth0 +ipv4.routes "192.168.0.0/24 via 192.168.0.1"

      Copy to Clipboard Toggle word wrap

   4. Apply the changes by running the following command:

      # nmcli connection up <interface_name>

      Copy to Clipboard Toggle word wrap

      Replace <interface_name> with the interface name.

   5. Verify the routing table to ensure the route has been added successfully:

      # ip route

      Copy to Clipboard Toggle word wrap

   6. Repeat the previous steps for each control plane node in the first subnet.

      Note

      Adjust the commands to match your actual interface names and gateway.

2. Configure the second subnet to communicate with the first subnet:

   1. Log in as root to a remote compute node by running the following command:

      $ sudo su -

      Copy to Clipboard Toggle word wrap

   2. Get the name of the network interface by running the following command:

      # nmcli dev status

      Copy to Clipboard Toggle word wrap

   3. Add a route to the first subnet (10.0.0.0) via the gateway by running the following command:

      # nmcli connection modify <interface_name> +ipv4.routes "10.0.0.0/24 via <gateway>"

      Copy to Clipboard Toggle word wrap

      Replace <interface_name> with the interface name. Replace <gateway> with the IP address of the actual gateway.

      Example

      # nmcli connection modify eth0 +ipv4.routes "10.0.0.0/24 via 10.0.0.1"

      Copy to Clipboard Toggle word wrap

   4. Apply the changes by running the following command:

      # nmcli connection up <interface_name>

      Copy to Clipboard Toggle word wrap

      Replace <interface_name> with the interface name.

   5. Verify the routing table to ensure the route has been added successfully by running the following command:

      # ip route

      Copy to Clipboard Toggle word wrap

   6. Repeat the previous steps for each compute node in the second subnet.

      Note

      Adjust the commands to match your actual interface names and gateway.

3. After you have configured the networks, test the connectivity to ensure the remote nodes can reach the control plane nodes and the control plane nodes can reach the remote nodes.

   1. From the control plane nodes in the first subnet, ping a remote node in the second subnet by running the following command:

      $ ping <remote_node_ip_address>

      Copy to Clipboard Toggle word wrap

      If the ping is successful, it means the control plane nodes in the first subnet can reach the remote nodes in the second subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.

   2. From the remote nodes in the second subnet, ping a control plane node in the first subnet by running the following command:

      $ ping <control_plane_node_ip_address>

      Copy to Clipboard Toggle word wrap

      If the ping is successful, it means the remote compute nodes in the second subnet can reach the control plane in the first subnet. If you do not receive a response, review the network configurations and repeat the procedure for the node.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install

   Copy to Clipboard Toggle word wrap

   $ export pullsecret_file=~/pull-secret.txt

   Copy to Clipboard Toggle word wrap

   $ export extract_dir=$(pwd)

   Copy to Clipboard Toggle word wrap

2. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

   Copy to Clipboard Toggle word wrap

3. Extract the installer:

   $ sudo cp oc /usr/local/bin

   Copy to Clipboard Toggle word wrap

   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}

   Copy to Clipboard Toggle word wrap

   $ sudo cp openshift-baremetal-install /usr/local/bin

   Copy to Clipboard Toggle word wrap

Procedure

1. Install podman:

   $ sudo dnf install -y podman

   Copy to Clipboard Toggle word wrap

2. Open firewall port 8080 to be used for RHCOS image caching:

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   Copy to Clipboard Toggle word wrap

   $ sudo firewall-cmd --reload

   Copy to Clipboard Toggle word wrap

3. Create a directory to store the bootstraposimage:

   $ mkdir /home/kni/rhcos_image_cache

   Copy to Clipboard Toggle word wrap

4. Set the appropriate SELinux context for the newly created directory:

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"

   Copy to Clipboard Toggle word wrap

   $ sudo restorecon -Rv /home/kni/rhcos_image_cache/

   Copy to Clipboard Toggle word wrap

5. Get the URI for the RHCOS image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_URI=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk.location')

   Copy to Clipboard Toggle word wrap

6. Get the name of the image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_NAME=${RHCOS_QEMU_URI##*/}

   Copy to Clipboard Toggle word wrap

7. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM:

   $ export RHCOS_QEMU_UNCOMPRESSED_SHA256=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk["uncompressed-sha256"]')

   Copy to Clipboard Toggle word wrap

8. Download the image and place it in the /home/kni/rhcos_image_cache directory:

   $ curl -L ${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_NAME}

   Copy to Clipboard Toggle word wrap

9. Confirm SELinux type is of httpd_sys_content_t for the new file:

   $ ls -Z /home/kni/rhcos_image_cache

   Copy to Clipboard Toggle word wrap

10. Create the pod:

    $ podman run -d --name rhcos_image_cache \

    1

    
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    registry.access.redhat.com/ubi9/httpd-24

    Copy to Clipboard Toggle word wrap

    1

    Creates a caching webserver with the name rhcos_image_cache. This pod serves the bootstrapOSImage image in the install-config.yaml file for deployment.

11. Generate the bootstrapOSImage configuration:

    $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)

    Copy to Clipboard Toggle word wrap

    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_NAME}?sha256=${RHCOS_QEMU_UNCOMPRESSED_SHA256}"

    Copy to Clipboard Toggle word wrap

    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"

    Copy to Clipboard Toggle word wrap

12. Add the required configuration to the install-config.yaml file under platform.baremetal:

    platform:
      baremetal:
        bootstrapOSImage: <bootstrap_os_image>  

    1

    Copy to Clipboard Toggle word wrap

    1

    Replace <bootstrap_os_image> with the value of $BOOTSTRAP_OS_IMAGE.

    See the "Configuring the install-config.yaml file" section for additional details.