Chapter 4. Installing a cluster with customizations

Procedure

1. Log in to the Red Hat Hybrid Cloud Console using your login credentials.
2. Navigate to Datacenter.
3. Click Run Agent-based Installer locally.
4. Select the operating system and architecture for the OpenShift Installer and Command line interface.
5. Click Download Installer to download and extract the install program.
6. Download or copy the pull secret by clicking on Download pull secret or Copy pull secret.
7. Click Download command-line tools and place the openshift-install binary in a directory that is on your PATH.

Procedure

1. Log in to the OpenShift CLI (oc).

2. Check your release payload by running the following command:

   $ ./openshift-install version

   Example output

   ./openshift-install 4.18.0
   built from commit abc123def456
   release image quay.io/openshift-release-dev/ocp-release@sha256:123abc456def789ghi012jkl345mno678pqr901stu234vwx567yz0
   release architecture amd64

   If you are using the release image with the multi payload, the release architecture displayed in the output of this command is the default architecture.

3. To check the architecture of the payload, run the following command:

   $ oc adm release info <release_image> -o jsonpath="{ .metadata.metadata}" 

   1

   1

   Replace <release_image> with the release image. For example: quay.io/openshift-release-dev/ocp-release@sha256:123abc456def789ghi012jkl345mno678pqr901stu234vwx567yz0.

   Example output when the release image uses the multi payload

   {"release.openshift.io architecture":"multi"}

   If you are using the release image with the multi payload, you can install the cluster on different architectures such as arm64, amd64, s390x, and ppc64le. Otherwise, you can install the cluster only on the release architecture displayed in the output of the openshift-install version command.

Procedure

1. Install the nmstate dependency by running the following command:

   $ sudo dnf install /usr/bin/nmstatectl -y

2. Place the openshift-install binary in a directory that is on your PATH.

3. Create a directory to store the install configuration by running the following command:

   $ mkdir ~/<directory_name>

4. Create the install-config.yaml file by running the following command:

   $ cat << EOF > ./<directory_name>/install-config.yaml
   apiVersion: v1
   baseDomain: test.example.com
   compute:
   - architecture: amd64 

   1

   
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 1
   metadata:
     name: sno-cluster 

   2

   
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     machineNetwork:
     - cidr: 192.168.0.0/16
     networkType: OVNKubernetes 

   3

   
     serviceNetwork:
     - 172.30.0.0/16
   platform: 

   4

   
     none: {}
   pullSecret: '<pull_secret>' 

   5

   
   sshKey: '<ssh_pub_key>' 

   6

   
   additionalTrustBundle: | 

   7

   
     -----BEGIN CERTIFICATE-----
     ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
     -----END CERTIFICATE-----
   imageContentSources: 

   8

   
   - mirrors:
     - <local_registry>/<local_repository_name>/release
     source: quay.io/openshift-release-dev/ocp-release
   - mirrors:
     - <local_registry>/<local_repository_name>/release
     source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
   EOF

   1

   Specify the system architecture. Valid values are amd64, arm64, ppc64le, and s390x.

   If you are using the release image with the multi payload, you can install the cluster on different architectures such as arm64, amd64, s390x, and ppc64le. Otherwise, you can install the cluster only on the release architecture displayed in the output of the openshift-install version command. For more information, see "Verifying the supported architecture for installing an Agent-based Installer cluster".

   2

   Required. Specify your cluster name.

   3

   The cluster network plugin to install. The default value OVNKubernetes is the only supported value.

   4

   Specify your platform.

   Note

   For bare-metal platforms, host settings made in the platform section of the install-config.yaml file are used by default, unless they are overridden by configurations made in the agent-config.yaml file.

   5

   Specify your pull secret.

   6

   Specify your SSH public key.

   7

   Provide the contents of the certificate file that you used for your mirror registry. The certificate file can be an existing, trusted certificate authority or the self-signed certificate that you generated for the mirror registry. You must specify this parameter if you are using a disconnected mirror registry.

   8

   Provide the imageContentSources section according to the output of the command that you used to mirror the repository. You must specify this parameter if you are using a disconnected mirror registry.

   Important

   • When using the oc adm release mirror command, use the output from the imageContentSources section.
   • When using the oc mirror command, use the repositoryDigestMirrors section of the ImageContentSourcePolicy file that results from running the command.
   • The ImageContentSourcePolicy resource is deprecated.

   Note

   If you set the platform to vSphere, baremetal, or none, you can configure IP address endpoints for cluster nodes in three ways:

   • IPv4
   • IPv6
   • IPv4 and IPv6 in parallel (dual-stack)

   Example of dual-stack networking

   networking:
     clusterNetwork:
       - cidr: 172.21.0.0/16
         hostPrefix: 23
       - cidr: fd02::/48
         hostPrefix: 64
     machineNetwork:
       - cidr: 192.168.11.0/16
       - cidr: 2001:DB8::/32
     serviceNetwork:
       - 172.22.0.0/16
       - fd03::/112
     networkType: OVNKubernetes
   platform:
     baremetal:
       apiVIPs:
       - 192.168.11.3
       - 2001:DB8::4
       ingressVIPs:
       - 192.168.11.4
       - 2001:DB8::5

   Note

   When you use a disconnected mirror registry, you must add the certificate file that you created previously for your mirror registry to the additionalTrustBundle field of the install-config.yaml file.

5. Create the agent-config.yaml file by running the following command:

   $ cat > agent-config.yaml << EOF
   apiVersion: v1beta1
   kind: AgentConfig
   metadata:
     name: sno-cluster
   rendezvousIP: 192.168.111.80 

   1

   
   hosts: 

   2

   
     - hostname: master-0 

   3

   
       interfaces:
         - name: eno1
           macAddress: 00:ef:44:21:e6:a5
       rootDeviceHints: 

   4

   
         deviceName: /dev/sdb
       networkConfig: 

   5

   
         interfaces:
           - name: eno1
             type: ethernet
             state: up
             mac-address: 00:ef:44:21:e6:a5
             ipv4:
               enabled: true
               address:
                 - ip: 192.168.111.80
                   prefix-length: 23
               dhcp: false
         dns-resolver:
           config:
             server:
               - 192.168.111.1
         routes:
           config:
             - destination: 0.0.0.0/0
               next-hop-address: 192.168.111.2
               next-hop-interface: eno1
               table-id: 254
   EOF

   1

   This IP address is used to determine which node performs the bootstrapping process as well as running the assisted-service component. You must provide the rendezvous IP address when you do not specify at least one host’s IP address in the networkConfig parameter. If this address is not provided, one IP address is selected from the provided hosts' networkConfig.

   2

   Optional: Host configuration. The number of hosts defined must not exceed the total number of hosts defined in the install-config.yaml file, which is the sum of the values of the compute.replicas and controlPlane.replicas parameters.

   3

   Optional: Overrides the hostname obtained from either the Dynamic Host Configuration Protocol (DHCP) or a reverse DNS lookup. Each host must have a unique hostname supplied by one of these methods.

   4

   Enables provisioning of the Red Hat Enterprise Linux CoreOS (RHCOS) image to a particular device. The installation program examines the devices in the order it discovers them, and compares the discovered values with the hint values. It uses the first discovered device that matches the hint value.

   5

   Optional: Configures the network interface of a host in NMState format.

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:
   - name: enp2s0
     type: ethernet
     state: up
     mtu: 9000
     ipv4:
       enabled: false
     ipv6:
       enabled: false
   - name: br-ex
     type: ovs-bridge
     state: up
     ipv4:
       enabled: false
       dhcp: false
     ipv6:
       enabled: false
       dhcp: false
     bridge:
       options:
         mcast-snooping-enable: true
       port:
       - name: enp2s0
       - name: br-ex
   - name: br-ex
     type: ovs-interface
     state: up
     copy-mac-from: enp2s0
     mtu: 9000
     ipv4:
       enabled: true
       dhcp: true
       auto-route-metric: 48
     ipv6:
       enabled: true
       dhcp: true
       auto-route-metric: 48
   # ...

   where:

   interfaces.name

   Specifies the name of the interface.

   interfaces.type

   Specifies the type of ethernet.

   interfaces.state

   Specifies the requested state for the interface after creation.

   mtu

   To ensure network stability and performance, you must explicitly declare the MTU in the manifest for every interface. Do not rely on automatic MTU configuration. The MTU configured on a bridge port or VLAN-tagged interface must not exceed the maximum frame size supported by the attached physical medium. A mismatch causes packet fragmentation or connectivity loss.

   ipv4.enabled

   Disables IPv4 and IPv6 in this example.

   port.name

   Specifies the node NIC to which the bridge attaches.

   auto-route-metric

   Sets the parameter to 48 to ensure the br-ex default route always has the highest precedence (lowest metric). This configuration prevents routing conflicts with any other interfaces that are automatically configured by the NetworkManager service.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yml | base64

   where:

   <nmstate_configuration>

   Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig file as an additional manifest file. Define a customized br-ex bridge network configuration analogous to the following example in the file. The Agent-based Installer automatically applies the updates from the MachineConfig object to your cluster.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 10-br-ex-worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration>
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/worker-0.yml
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration>
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/worker-1.yml
   # ...

   where:

   metadata.name

   Specifies the name of the policy.

   contents.source

   Writes the encoded base64 information to the specified path.

   path

   For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. The worker role is the default role for nodes in your cluster. You must use the .yml extension for configuration files. For example, use $(hostname -s).yml when specifying the short hostname path for each node or all nodes in the MachineConfig manifest file.

   You can apply a single global configuration to all nodes in your cluster by using the /etc/nmstate/openshift/cluster.yml configuration file. In this case, you do not need to specify the short hostname path for each node, such as /etc/nmstate/openshift/<node_hostname>.yml. For example:

   Example /etc/nmstate/openshift/cluster.yml configuration file

   # ...
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration>
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml
   # ...

4. Save the additional manifest file in the openshift subdirectory of your installation directory.

   On completing other configuration inputs for your installation, such as encrypting the disk, you create the ISO image. After booting this image, the customized br-ex bridge configuration applies to each node in your cluster.

Procedure

1. Generate ZTP cluster manifests by running the following command:

   $ openshift-install agent create cluster-manifests --dir <installation_directory>

   Important

   If you have created the install-config.yaml and agent-config.yaml files, those files are deleted and replaced by the cluster manifests generated through this command.

   Any configurations made to the install-config.yaml and agent-config.yaml files are imported to the ZTP cluster manifests when you run the openshift-install agent create cluster-manifests command.

2. Navigate to the cluster-manifests directory by running the following command:

   $ cd <installation_directory>/cluster-manifests

3. Configure the manifest files in the cluster-manifests directory. For sample files, see the "Sample GitOps ZTP custom resources" section.

4. Disconnected clusters: If you did not define mirror configuration in the install-config.yaml file before generating the ZTP manifests, perform the following steps:

   1. Navigate to the mirror directory by running the following command:

      $ cd ../mirror

   2. Configure the manifest files in the mirror directory.

Procedure

1. Generate ZTP cluster manifests by running the following command:

   $ openshift-install agent create cluster-manifests --dir <installation_directory>

   Important

   If you have created the install-config.yaml and agent-config.yaml files, those files are deleted and replaced by the cluster manifests generated through this command.

   Any configurations made to the install-config.yaml and agent-config.yaml files are imported to the ZTP cluster manifests when you run the openshift-install agent create cluster-manifests command.

   Note

   If you have already generated ZTP manifests, skip this step.

2. Navigate to the cluster-manifests directory by running the following command:

   $ cd <installation_directory>/cluster-manifests

3. Add the following section to the agent-cluster-install.yaml file:

   diskEncryption:
       enableOn: all 

   1

   
       mode: tang 

   2

   
       tangServers: "server1": "http://tang-server-1.example.com:7500" 

   3

   1

   Specify which nodes to enable disk encryption on. Valid values are none, all, masters, and workers.

   2

   Specify which disk encryption mode to use. Valid values are tpmv2 and tang.

   3

   Optional: If you are using Tang, specify the Tang servers.

Procedure

1. Create the agent image by running the following command:

   $ openshift-install --dir <install_directory> agent create image

   Note

   Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with a default /etc/multipath.conf configuration.

2. If you plan to boot the ISO image from a USB drive, add a master boot record to the image by running the following command:

   $ isohybrid --uefi <agent_iso_image>

   Example command

   $ isohybrid --uefi agent.x86_64.iso

3. Boot the agent.x86_64.iso, agent.aarch64.iso, or agent.s390x.iso image on the bare-metal machines.

Procedure

1. Boot your RHEL KVM machine.

2. To deploy the virtual server, run the virt-install command with the following parameters:

   ISO boot

   $ virt-install
       --name <vm_name> \
       --autostart \
       --memory=<memory> \
       --cpu host \
       --vcpus=<vcpus> \
       --cdrom \<path_to_image>/<agent_iso_image> \ 

   1

   
       --disk pool=default,size=<disk_pool_size> \
       --network network:default,mac=<mac_address> \
       --graphics none \
       --noautoconsole \
       --os-variant rhel9.0 \
       --wait=-1

   1

   For the --cdrom parameter, specify the location of the ISO image on the local server, for example, <path_to_image>/home/<image>.iso.

3. Optional: Enable FIPS mode.

   To enable FIPS mode on IBM Z® clusters with RHEL KVM you must use PXE boot instead and run the virt-install command with the following parameters:

   PXE boot

   $ virt-install \
      --name <vm_name> \
      --autostart \
      --ram=16384 \
      --cpu host \
      --vcpus=8 \
      --location <path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img \

   1

   
      --disk <qcow_image_path> \
      --network network:macvtap ,mac=<mac_address> \
      --graphics none \
      --noautoconsole \
      --wait=-1 \
      --extra-args "rd.neednet=1 nameserver=<nameserver>" \
      --extra-args "ip=<IP>::<nameserver>::<hostname>:enc1:none" \
      --extra-args "coreos.live.rootfs_url=http://<http_server>:8080/agent.s390x-rootfs.img" \
      --extra-args "random.trust_cpu=on rd.luks.options=discard" \
      --extra-args "ignition.firstboot ignition.platform.id=metal" \
      --extra-args "console=tty1 console=ttyS1,115200n8" \
      --extra-args "coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" \
      --extra-args "fips=1" \

   2

   
      --osinfo detect=on,require=off

   1

   For the --location parameter, specify the location of the kernel/initrd on the HTTP or HTTPS server.

   2

   To enable FIPS mode, specify fips=1. This entry is required in addition to setting the fips parameter to true in the install-config.yaml file.

   Note

   Currently, only PXE boot is supported to enable FIPS mode on IBM Z®.

Procedure

1. Wait for the agent console application to check whether or not the configured release image can be pulled from a registry.

2. If the agent console application states that the installer connectivity checks have passed, wait for the prompt to time out to continue with the installation.

   Note

   You can still choose to view or change network configuration settings even if the connectivity checks have passed.

   However, if you choose to interact with the agent console application rather than letting it time out, you must manually quit the TUI to proceed with the installation.

3. If the agent console application checks have failed, which is indicated by a red icon beside the Release image URL pull check, use the following steps to reconfigure the host’s network settings:

   1. Read the Check Errors section of the TUI. This section displays error messages specific to the failed checks.

      
   2. Select Configure network to launch the NetworkManager TUI.
   3. Select Edit a connection and select the connection you want to reconfigure.
   4. Edit the configuration and select OK to save your changes.
   5. Select Back to return to the main screen of the NetworkManager TUI.
   6. Select Activate a Connection.
   7. Select the reconfigured network to deactivate it.
   8. Select the reconfigured network again to reactivate it.
   9. Select Back and then select Quit to return to the agent console application.
   10. Wait at least five seconds for the continuous network checks to restart using the new network configuration.
   11. If the Release image URL pull check succeeds and displays a green icon beside the URL, select Quit to exit the agent console application and continue with the installation.

Procedure

1. Optional: To know when the bootstrap host (rendezvous host) reboots, run the following command:

   $ ./openshift-install --dir <install_directory> agent wait-for bootstrap-complete \

   1

   
       --log-level=info 

   2

   1

   For <install_directory>, specify the path to the directory where the agent ISO was generated.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Example output

   ...................................................................
   ...................................................................
   INFO Bootstrap configMap status is complete
   INFO cluster bootstrap is complete

   The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

2. To track the progress and verify successful installation, run the following command:

   $ openshift-install --dir <install_directory> agent wait-for install-complete 

   1

   1

   For <install_directory> directory, specify the path to the directory where the agent ISO was generated.

   Example output

   ...................................................................
   ...................................................................
   INFO Cluster is installed
   INFO Install complete!
   INFO To access the cluster as the system:admin user when using 'oc', run
   INFO     export KUBECONFIG=/home/core/installer/auth/kubeconfig
   INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sno-cluster.test.example.com