Installing on bare metal

1. One CPU is equivalent to one physical core when simultaneous multithreading (SMT), or Hyper-Threading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = CPUs.
2. OpenShift Container Platform and Kubernetes are sensitive to disk performance, and faster storage is recommended, particularly for etcd on the control plane nodes which require a 10 ms p99 fsync duration. Note that on many cloud platforms, storage size and IOPS scale together, so you might need to over-allocate storage volume to obtain sufficient performance.
3. As with all user-provisioned installations, if you choose to use RHEL compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks. Use of RHEL 7 compute machines is deprecated and has been removed in OpenShift Container Platform 4.10 and later.

1. API load balancer: Provides a common endpoint for users, both human and machine, to interact with and configure the platform. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A stateless load balancing algorithm. The options vary based on the load balancer implementation.
   Important

   Do not configure session persistence for an API load balancer. Configuring session persistence for a Kubernetes API server might cause performance issues from excess application traffic for your OpenShift Container Platform cluster and the Kubernetes API that runs inside the cluster.

   Configure the following ports on both the front and back of the load balancers:

   Table 2.7. API load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   6443	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane. You must configure the /readyz endpoint for the API server health check probe.	X	X	Kubernetes API server
   22623	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	X		Machine config server

   Note

   The load balancer must be configured to take a maximum of 30 seconds from the time the API server turns off the /readyz endpoint to the removal of the API server instance from the pool. Within the time frame after /readyz returns an error or becomes healthy, the endpoint must have been removed or added. Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values.

2. Application Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. A working configuration for the Ingress router is required for an OpenShift Container Platform cluster.

   Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A connection-based or session-based persistence is recommended, based on the options available and types of applications that will be hosted on the platform.
   Tip

   If the true IP address of the client can be seen by the application Ingress load balancer, enabling source IP-based session persistence can improve performance for applications that use end-to-end TLS encryption.

   Configure the following ports on both the front and back of the load balancers:

   Table 2.8. Application Ingress load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   443	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTPS traffic
   80	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTP traffic

   Note

   If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods run on the control plane nodes. In three-node cluster deployments, you must configure your application Ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes.

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:
   - name: enp2s0 1
     type: ethernet 2
     state: up 3
     ipv4:
       enabled: false 4
     ipv6:
       enabled: false
   - name: br-ex
     type: ovs-bridge
     state: up
     ipv4:
       enabled: false
       dhcp: false
     ipv6:
       enabled: false
       dhcp: false
     bridge:
       port:
       - name: enp2s0 5
       - name: br-ex
   - name: br-ex
     type: ovs-interface
     state: up
     copy-mac-from: enp2s0
     ipv4:
       enabled: true
       dhcp: true
     ipv6:
       enabled: false
       dhcp: false
   # ...

   1

   Name of the interface.

   2

   The type of ethernet.

   3

   The requested state for the interface after creation.

   4

   Disables IPv4 and IPv6 in this example.

   5

   The node NIC to which the bridge attaches.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yaml | base64 1

   1

   Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig manifest file and define a customized br-ex bridge network configuration analogous to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 1
     name: 10-br-ex-worker 2
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 3
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml
   # ...

   1

   For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. If you have a single global configuration specified in an /etc/nmstate/openshift/cluster.yml configuration file that you want to apply to all nodes in your cluster, you do not need to specify the hostname path for each node. The worker role is the default role for nodes in your cluster. The .yaml extension does not work when specifying the hostname path for each node or all nodes in the MachineConfig manifest file.

   2

   The name of the policy.

   3

   Writes the encoded base64 information to the specified path.

Procedure

1. Edit the MachineConfig CR by entering the following command:

   $ oc edit mc <machineconfig_custom_resource_name>

2. Add each compute node configuration to the CR, so that the CR can manage roles for each defined compute node in your cluster.
3. Create a Secret object named extraworker-secret that has a minimal static IP configuration.

4. Apply the extraworker-secret secret to each node in your cluster by entering the following command. This step provides each compute node access to the Ignition config file.

   $ oc apply -f ./extraworker-secret.yaml

5. Create a BareMetalHost resource and specify the network secret in the preprovisioningNetworkDataName parameter:

   Example BareMetalHost resource with an attached network secret

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   spec:
   # ...
     preprovisioningNetworkDataName: ostest-extraworker-0-network-config-secret
   # ...

6. To manage the BareMetalHost object within the openshift-machine-api namespace of your cluster, change to the namespace by entering the following command:

   $ oc project openshift-machine-api

7. Get the machine sets:

   $ oc get machinesets

8. Scale each machine set by entering the following command. You must run this command for each machine set.

   $ oc scale machineset <machineset_name> --replicas=<n> 1

   1

   Where <machineset_name> is the name of the machine set and <n> is the number of compute nodes.

Procedure

1. Edit your install-config.yaml file and add the proxy settings. For example:

   apiVersion: v1
   baseDomain: my.domain.com
   proxy:
     httpProxy: http://<username>:<pswd>@<ip>:<port> 1
     httpsProxy: https://<username>:<pswd>@<ip>:<port> 2
     noProxy: example.com 3
   additionalTrustBundle: | 4
       -----BEGIN CERTIFICATE-----
       <MY_TRUSTED_CA_CERT>
       -----END CERTIFICATE-----
   additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> 5

   1

   A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   2

   A proxy URL to use for creating HTTPS connections outside the cluster.

   3

   A comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all destinations.

   4

   If provided, the installation program generates a config map that is named user-ca-bundle in the openshift-config namespace that contains one or more additional CA certificates that are required for proxying HTTPS connections. The Cluster Network Operator then creates a trusted-ca-bundle config map that merges these contents with the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle, and this config map is referenced in the trustedCA field of the Proxy object. The additionalTrustBundle field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

   5

   Optional: The policy to determine the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always. Use Proxyonly to reference the user-ca-bundle config map only when http/https proxy is configured. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly.

   Note

   The installation program does not support the proxy readinessEndpoints field.

   Note

   If the installer times out, restart and then complete the deployment by using the wait-for command of the installer. For example:

   $ ./openshift-install wait-for install-complete --log-level debug

2. Save the file and reference it when installing OpenShift Container Platform.

Procedure

1. Obtain the SHA512 digest for each of your Ignition config files. For example, you can use the following on a system running Linux to get the SHA512 digest for your bootstrap.ign Ignition config file:

   $ sha512sum <installation_directory>/bootstrap.ign

   The digests are provided to the coreos-installer in a later step to validate the authenticity of the Ignition config files on the cluster nodes.

2. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   Important

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

3. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign 1

   Example output

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

4. Although it is possible to obtain the RHCOS images that are required for your preferred method of installing operating system instances from the RHCOS image mirror page, the recommended way to obtain the correct version of your RHCOS images are from the output of openshift-install command:

   $ openshift-install coreos print-stream-json | grep '\.iso[^.]'

   Example output

   "location": "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live.aarch64.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live.ppc64le.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live.s390x.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live.x86_64.iso",

   Important

   The RHCOS images might not change with every release of OpenShift Container Platform. You must download images with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image versions that match your OpenShift Container Platform version if they are available. Use only ISO images for this procedure. RHCOS qcow2 images are not supported for this installation type.

   ISO file names resemble the following example:

   rhcos-<version>-live.<architecture>.iso

5. Use the ISO to start the RHCOS installation. Use one of the following installation options:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection by using a lights-out management (LOM) interface.

6. Boot the RHCOS ISO image without specifying any options or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   It is possible to interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you should use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1 1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Be sure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. After RHCOS installs, you must reboot the system. During the system reboot, it applies the Ignition config file that you specified.

10. Check the console output to verify that Ignition ran.

    Example command

    Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)
    Ignition: user-provided config was applied

11. Continue to create the other machines for your cluster.

    Important

    You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install OpenShift Container Platform.

    If the required network, DNS, and load balancer infrastructure are in place, the OpenShift Container Platform bootstrap process begins automatically after the RHCOS nodes have rebooted.

    Note

    RHCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OpenShift Container Platform 4 cluster nodes running RHCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

Procedure

1. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   Important

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

2. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign 1

   Example output

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

3. Although it is possible to obtain the RHCOS kernel, initramfs and rootfs files that are required for your preferred method of installing operating system instances from the RHCOS image mirror page, the recommended way to obtain the correct version of your RHCOS files are from the output of openshift-install command:

   $ openshift-install coreos print-stream-json | grep -Eo '"https.*(kernel-|initramfs.|rootfs.)\w+(\.img)?"'

   Example output

   "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-kernel-aarch64"
   "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-initramfs.aarch64.img"
   "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-rootfs.aarch64.img"
   "<url>/art/storage/releases/rhcos-4.17-ppc64le/49.84.202110081256-0/ppc64le/rhcos-<release>-live-kernel-ppc64le"
   "<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live-initramfs.ppc64le.img"
   "<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live-rootfs.ppc64le.img"
   "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-kernel-s390x"
   "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-initramfs.s390x.img"
   "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-rootfs.s390x.img"
   "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-kernel-x86_64"
   "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-initramfs.x86_64.img"
   "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-rootfs.x86_64.img"

   Important

   The RHCOS artifacts might not change with every release of OpenShift Container Platform. You must download images with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Only use the appropriate kernel, initramfs, and rootfs artifacts described below for this procedure. RHCOS QCOW2 images are not supported for this installation type.

   The file names contain the OpenShift Container Platform version number. They resemble the following examples:

   • kernel: rhcos-<version>-live-kernel-<architecture>
   • initramfs: rhcos-<version>-live-initramfs.<architecture>.img
   • rootfs: rhcos-<version>-live-rootfs.<architecture>.img

4. Upload the rootfs, kernel, and initramfs files to your HTTP server.

   Important

   If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

5. Configure the network boot infrastructure so that the machines boot from their local disks after RHCOS is installed on them.

6. Configure PXE or iPXE installation for the RHCOS images and begin the installation.

   Modify one of the following example menu entries for your environment and verify that the image and Ignition files are properly accessible:

   • For PXE (x86_64):

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign 2 3

     1 1

     Specify the location of the live kernel file that you uploaded to your HTTP server. The URL must be HTTP, TFTP, or FTP; HTTPS and NFS are not supported.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the initramfs file, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file. You can also add more kernel arguments to the APPEND line to configure networking or other boot options.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

   • For iPXE (x86_64 + aarch64 ):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and Grub as second stage) on aarch64:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

7. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Be sure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

8. After RHCOS installs, the system reboots. During reboot, the system applies the Ignition config file that you specified.

9. Check the console output to verify that Ignition ran.

   Example command

   Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)
   Ignition: user-provided config was applied

10. Continue to create the machines for your cluster.

    Important

    You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install the cluster.

    If the required network, DNS, and load balancer infrastructure are in place, the OpenShift Container Platform bootstrap process begins automatically after the RHCOS nodes have rebooted.

    Note

    RHCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OpenShift Container Platform 4 cluster nodes running RHCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

Procedure

1. To enable multipath and start the multipathd daemon, run the following command on the installation host:

   $ mpathconf --enable && systemctl start multipathd.service

   • Optional: If booting the PXE or ISO, you can instead enable multipath by adding rd.multipath=default from the kernel command line.

2. Append the kernel arguments by invoking the coreos-installer program:

   • If there is only one multipath device connected to the machine, it should be available at path /dev/mapper/mpatha. For example:

     $ coreos-installer install /dev/mapper/mpatha \ 1
     --append-karg rd.multipath=default \
     --append-karg root=/dev/disk/by-label/dm-mpath-root \
     --append-karg rw

     1

     Indicates the path of the single multipathed device.

   • If there are multiple multipath devices connected to the machine, or to be more explicit, instead of using /dev/mapper/mpatha, it is recommended to use the World Wide Name (WWN) symlink available in /dev/disk/by-id. For example:

     $ coreos-installer install /dev/disk/by-id/wwn-<wwn_ID> \ 1
     --append-karg rd.multipath=default \
     --append-karg root=/dev/disk/by-label/dm-mpath-root \
     --append-karg rw

     1

     Indicates the WWN ID of the target multipathed device. For example, 0xx194e957fcedb4841.

     This symlink can also be used as the coreos.inst.install_dev kernel argument when using special coreos.inst.* arguments to direct the live installer. For more information, see "Installing RHCOS and starting the OpenShift Container Platform bootstrap process".

3. Check that the kernel arguments worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   ...
   rd.multipath=default root=/dev/disk/by-label/dm-mpath-root
   ...
   
   sh-4.2# exit

   You should see the added kernel arguments.

Prerequisites

1. You are in the RHCOS live environment.
2. You have an iSCSI target that you want to install RHCOS on.

Procedure

1. Mount the iSCSI target from the live environment by running the following command:

   $ iscsiadm \
       --mode discovery \
       --type sendtargets
       --portal <IP_address> \ 1
       --login

   1

   The IP address of the target portal.

2. Install RHCOS onto the iSCSI target by running the following command and using the necessary kernel arguments, for example:

   $ coreos-installer install \
       /dev/disk/by-path/ip-<IP_address>:<port>-iscsi-<target_iqn>-lun-<lun> \ 1
       --append-karg rd.iscsi.initiator=<initiator_iqn> \ 2
       --append.karg netroot=<target_iqn> \ 3
       --console ttyS0,115200n8
       --ignition-file <path_to_file>

   1

   The location you are installing to. You must provide the IP address of the target portal, the associated port number, the target iSCSI node in IQN format, and the iSCSI logical unit number (LUN).

   2

   The iSCSI initiator, or client, name in IQN format. The initiator forms a session to connect to the iSCSI target.

   3

   The the iSCSI target, or server, name in IQN format.

   For more information about the iSCSI options supported by dracut, see the dracut.cmdline manual page.

3. Unmount the iSCSI disk with the following command:

   $ iscsiadm --mode node --logoutall=all

Prerequisites

1. You are in the RHCOS live environment.
2. You have an iSCSI target you want to install RHCOS on.
3. Optional: you have multipathed your iSCSI target.

Procedure

1. Mount the iSCSI target from the live environment by running the following command:

   $ iscsiadm \
       --mode discovery \
       --type sendtargets
       --portal <IP_address> \ 1
       --login

   1

   The IP address of the target portal.

2. Optional: enable multipathing and start the daemon with the following command:

   $ mpathconf --enable && systemctl start multipathd.service

3. Install RHCOS onto the iSCSI target by running the following command and using the necessary kernel arguments, for example:

   $ coreos-installer install \
       /dev/mapper/mpatha \ 1
       --append-karg rd.iscsi.firmware=1 \ 2
       --append-karg rd.multipath=default \ 3
       --console ttyS0 \
       --ignition-file <path_to_file>

   1

   The path of a single multipathed device. If there are multiple multipath devices connected, or to be explicit, you can use the World Wide Name (WWN) symlink available in /dev/disk/by-path.

   2

   The iSCSI parameter is read from the BIOS firmware.

   3

   Optional: include this parameter if you are enabling multipathing.

   For more information about the iSCSI options supported by dracut, see the dracut.cmdline manual page.

4. Unmount the iSCSI disk:

   $ iscsiadm --mode node --logout=all

1. One CPU is equivalent to one physical core when simultaneous multithreading (SMT), or Hyper-Threading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = CPUs.
2. OpenShift Container Platform and Kubernetes are sensitive to disk performance, and faster storage is recommended, particularly for etcd on the control plane nodes which require a 10 ms p99 fsync duration. Note that on many cloud platforms, storage size and IOPS scale together, so you might need to over-allocate storage volume to obtain sufficient performance.
3. As with all user-provisioned installations, if you choose to use RHEL compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks. Use of RHEL 7 compute machines is deprecated and has been removed in OpenShift Container Platform 4.10 and later.

1. API load balancer: Provides a common endpoint for users, both human and machine, to interact with and configure the platform. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A stateless load balancing algorithm. The options vary based on the load balancer implementation.
   Important

   Do not configure session persistence for an API load balancer. Configuring session persistence for a Kubernetes API server might cause performance issues from excess application traffic for your OpenShift Container Platform cluster and the Kubernetes API that runs inside the cluster.

   Configure the following ports on both the front and back of the load balancers:

   Table 3.7. API load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   6443	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane. You must configure the /readyz endpoint for the API server health check probe.	X	X	Kubernetes API server
   22623	Bootstrap and control plane. You remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	X		Machine config server

   Note

   The load balancer must be configured to take a maximum of 30 seconds from the time the API server turns off the /readyz endpoint to the removal of the API server instance from the pool. Within the time frame after /readyz returns an error or becomes healthy, the endpoint must have been removed or added. Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values.

2. Application Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. A working configuration for the Ingress router is required for an OpenShift Container Platform cluster.

   Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.
   • A connection-based or session-based persistence is recommended, based on the options available and types of applications that will be hosted on the platform.
   Tip

   If the true IP address of the client can be seen by the application Ingress load balancer, enabling source IP-based session persistence can improve performance for applications that use end-to-end TLS encryption.

   Configure the following ports on both the front and back of the load balancers:

   Table 3.8. Application Ingress load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   443	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTPS traffic
   80	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTP traffic

   Note

   If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods run on the control plane nodes. In three-node cluster deployments, you must configure your application Ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes.

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:
   - name: enp2s0 1
     type: ethernet 2
     state: up 3
     ipv4:
       enabled: false 4
     ipv6:
       enabled: false
   - name: br-ex
     type: ovs-bridge
     state: up
     ipv4:
       enabled: false
       dhcp: false
     ipv6:
       enabled: false
       dhcp: false
     bridge:
       port:
       - name: enp2s0 5
       - name: br-ex
   - name: br-ex
     type: ovs-interface
     state: up
     copy-mac-from: enp2s0
     ipv4:
       enabled: true
       dhcp: true
     ipv6:
       enabled: false
       dhcp: false
   # ...

   1

   Name of the interface.

   2

   The type of ethernet.

   3

   The requested state for the interface after creation.

   4

   Disables IPv4 and IPv6 in this example.

   5

   The node NIC to which the bridge attaches.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yaml | base64 1

   1

   Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig manifest file and define a customized br-ex bridge network configuration analogous to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 1
     name: 10-br-ex-worker 2
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 3
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml
   # ...

   1

   For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. If you have a single global configuration specified in an /etc/nmstate/openshift/cluster.yml configuration file that you want to apply to all nodes in your cluster, you do not need to specify the hostname path for each node. The worker role is the default role for nodes in your cluster. The .yaml extension does not work when specifying the hostname path for each node or all nodes in the MachineConfig manifest file.

   2

   The name of the policy.

   3

   Writes the encoded base64 information to the specified path.

Procedure

1. Edit the MachineConfig CR by entering the following command:

   $ oc edit mc <machineconfig_custom_resource_name>

2. Add each compute node configuration to the CR, so that the CR can manage roles for each defined compute node in your cluster.
3. Create a Secret object named extraworker-secret that has a minimal static IP configuration.

4. Apply the extraworker-secret secret to each node in your cluster by entering the following command. This step provides each compute node access to the Ignition config file.

   $ oc apply -f ./extraworker-secret.yaml

5. Create a BareMetalHost resource and specify the network secret in the preprovisioningNetworkDataName parameter:

   Example BareMetalHost resource with an attached network secret

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   spec:
   # ...
     preprovisioningNetworkDataName: ostest-extraworker-0-network-config-secret
   # ...

6. To manage the BareMetalHost object within the openshift-machine-api namespace of your cluster, change to the namespace by entering the following command:

   $ oc project openshift-machine-api

7. Get the machine sets:

   $ oc get machinesets

8. Scale each machine set by entering the following command. You must run this command for each machine set.

   $ oc scale machineset <machineset_name> --replicas=<n> 1

   1

   Where <machineset_name> is the name of the machine set and <n> is the number of compute nodes.

Procedure

1. Obtain the SHA512 digest for each of your Ignition config files. For example, you can use the following on a system running Linux to get the SHA512 digest for your bootstrap.ign Ignition config file:

   $ sha512sum <installation_directory>/bootstrap.ign

   The digests are provided to the coreos-installer in a later step to validate the authenticity of the Ignition config files on the cluster nodes.

2. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   Important

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

3. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign 1

   Example output

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

4. Although it is possible to obtain the RHCOS images that are required for your preferred method of installing operating system instances from the RHCOS image mirror page, the recommended way to obtain the correct version of your RHCOS images are from the output of openshift-install command:

   $ openshift-install coreos print-stream-json | grep '\.iso[^.]'

   Example output

   "location": "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live.aarch64.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live.ppc64le.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live.s390x.iso",
   "location": "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live.x86_64.iso",

   Important

   The RHCOS images might not change with every release of OpenShift Container Platform. You must download images with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image versions that match your OpenShift Container Platform version if they are available. Use only ISO images for this procedure. RHCOS qcow2 images are not supported for this installation type.

   ISO file names resemble the following example:

   rhcos-<version>-live.<architecture>.iso

5. Use the ISO to start the RHCOS installation. Use one of the following installation options:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection by using a lights-out management (LOM) interface.

6. Boot the RHCOS ISO image without specifying any options or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   It is possible to interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you should use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1 1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Be sure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. After RHCOS installs, you must reboot the system. During the system reboot, it applies the Ignition config file that you specified.

10. Check the console output to verify that Ignition ran.

    Example command

    Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)
    Ignition: user-provided config was applied

11. Continue to create the other machines for your cluster.

    Important

    You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install OpenShift Container Platform.

    If the required network, DNS, and load balancer infrastructure are in place, the OpenShift Container Platform bootstrap process begins automatically after the RHCOS nodes have rebooted.

    Note

    RHCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OpenShift Container Platform 4 cluster nodes running RHCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.