Chapter 4. Installing a cluster with customizations

Procedure

1. Log in to the OpenShift Container Platform web console using your login credentials.
2. Navigate to Datacenter.
3. Click Run Agent-based Installer locally.
4. Select the operating system and architecture for the OpenShift Installer and Command line interface.
5. Click Download Installer to download and extract the install program.
6. Download or copy the pull secret by clicking on Download pull secret or Copy pull secret.
7. Click Download command-line tools and place the openshift-install binary in a directory that is on your PATH.

Procedure

1. Log in to the OpenShift CLI (oc).

2. Check your release payload by running the following command:

   $ ./openshift-install version

   Copy to Clipboard Toggle word wrap

   Example output

   ./openshift-install 4.18.0
   built from commit abc123def456
   release image quay.io/openshift-release-dev/ocp-release@sha256:123abc456def789ghi012jkl345mno678pqr901stu234vwx567yz0
   release architecture amd64

   Copy to Clipboard Toggle word wrap

   If you are using the release image with the multi payload, the release architecture displayed in the output of this command is the default architecture.

3. To check the architecture of the payload, run the following command:

   $ oc adm release info <release_image> -o jsonpath="{ .metadata.metadata}" 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <release_image> with the release image. For example: quay.io/openshift-release-dev/ocp-release@sha256:123abc456def789ghi012jkl345mno678pqr901stu234vwx567yz0.

   Example output when the release image uses the multi payload

   {"release.openshift.io architecture":"multi"}

   Copy to Clipboard Toggle word wrap

   If you are using the release image with the multi payload, you can install the cluster on different architectures such as arm64, amd64, s390x, and ppc64le. Otherwise, you can install the cluster only on the release architecture displayed in the output of the openshift-install version command.

Procedure

1. Install the nmstate dependency by running the following command:

   $ sudo dnf install /usr/bin/nmstatectl -y

   Copy to Clipboard Toggle word wrap
2. Place the openshift-install binary in a directory that is on your PATH.

3. Create a directory to store the install configuration by running the following command:

   $ mkdir ~/<directory_name>

   Copy to Clipboard Toggle word wrap

4. Create the install-config.yaml file by running the following command:

   $ cat << EOF > ./<directory_name>/install-config.yaml
   apiVersion: v1
   baseDomain: test.example.com
   compute:
   - architecture: amd64 

   1

   
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 1
   metadata:
     name: sno-cluster 

   2

   
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     machineNetwork:
     - cidr: 192.168.0.0/16
     networkType: OVNKubernetes 

   3

   
     serviceNetwork:
     - 172.30.0.0/16
   platform: 

   4

   
     none: {}
   pullSecret: '<pull_secret>' 

   5

   
   sshKey: '<ssh_pub_key>' 

   6

   
   EOF

   Copy to Clipboard Toggle word wrap

   1

   Specify the system architecture. Valid values are amd64, arm64, ppc64le, and s390x.

   If you are using the release image with the multi payload, you can install the cluster on different architectures such as arm64, amd64, s390x, and ppc64le. Otherwise, you can install the cluster only on the release architecture displayed in the output of the openshift-install version command. For more information, see "Verifying the supported architecture for installing an Agent-based Installer cluster".

   2

   Required. Specify your cluster name.

   3

   The cluster network plugin to install. The default value OVNKubernetes is the only supported value.

   4

   Specify your platform.

   Note

   For bare-metal platforms, host settings made in the platform section of the install-config.yaml file are used by default, unless they are overridden by configurations made in the agent-config.yaml file.

   5

   Specify your pull secret.

   6

   Specify your SSH public key.

   Note

   If you set the platform to vSphere, baremetal, or none, you can configure IP address endpoints for cluster nodes in three ways:

   • IPv4
   • IPv6
   • IPv4 and IPv6 in parallel (dual-stack)

   Example of dual-stack networking

   networking:
     clusterNetwork:
       - cidr: 172.21.0.0/16
         hostPrefix: 23
       - cidr: fd02::/48
         hostPrefix: 64
     machineNetwork:
       - cidr: 192.168.11.0/16
       - cidr: 2001:DB8::/32
     serviceNetwork:
       - 172.22.0.0/16
       - fd03::/112
     networkType: OVNKubernetes
   platform:
     baremetal:
       apiVIPs:
       - 192.168.11.3
       - 2001:DB8::4
       ingressVIPs:
       - 192.168.11.4
       - 2001:DB8::5

   Copy to Clipboard Toggle word wrap

   Note

   When you use a disconnected mirror registry, you must add the certificate file that you created previously for your mirror registry to the additionalTrustBundle field of the install-config.yaml file.

5. Create the agent-config.yaml file by running the following command:

   $ cat > agent-config.yaml << EOF
   apiVersion: v1beta1
   kind: AgentConfig
   metadata:
     name: sno-cluster
   rendezvousIP: 192.168.111.80 

   1

   
   hosts: 

   2

   
     - hostname: master-0 

   3

   
       interfaces:
         - name: eno1
           macAddress: 00:ef:44:21:e6:a5
       rootDeviceHints: 

   4

   
         deviceName: /dev/sdb
       networkConfig: 

   5

   
         interfaces:
           - name: eno1
             type: ethernet
             state: up
             mac-address: 00:ef:44:21:e6:a5
             ipv4:
               enabled: true
               address:
                 - ip: 192.168.111.80
                   prefix-length: 23
               dhcp: false
         dns-resolver:
           config:
             server:
               - 192.168.111.1
         routes:
           config:
             - destination: 0.0.0.0/0
               next-hop-address: 192.168.111.2
               next-hop-interface: eno1
               table-id: 254
   EOF

   Copy to Clipboard Toggle word wrap

   1

   This IP address is used to determine which node performs the bootstrapping process as well as running the assisted-service component. You must provide the rendezvous IP address when you do not specify at least one host’s IP address in the networkConfig parameter. If this address is not provided, one IP address is selected from the provided hosts' networkConfig.

   2

   Optional: Host configuration. The number of hosts defined must not exceed the total number of hosts defined in the install-config.yaml file, which is the sum of the values of the compute.replicas and controlPlane.replicas parameters.

   3

   Optional: Overrides the hostname obtained from either the Dynamic Host Configuration Protocol (DHCP) or a reverse DNS lookup. Each host must have a unique hostname supplied by one of these methods.

   4

   Enables provisioning of the Red Hat Enterprise Linux CoreOS (RHCOS) image to a particular device. The installation program examines the devices in the order it discovers them, and compares the discovered values with the hint values. It uses the first discovered device that matches the hint value.

   5

   Optional: Configures the network interface of a host in NMState format.

Procedure

1. Generate ZTP cluster manifests by running the following command:

   $ openshift-install agent create cluster-manifests --dir <installation_directory>

   Copy to Clipboard Toggle word wrap
   Important

   If you have created the install-config.yaml and agent-config.yaml files, those files are deleted and replaced by the cluster manifests generated through this command.

   Any configurations made to the install-config.yaml and agent-config.yaml files are imported to the ZTP cluster manifests when you run the openshift-install agent create cluster-manifests command.

2. Navigate to the cluster-manifests directory by running the following command:

   $ cd <installation_directory>/cluster-manifests

   Copy to Clipboard Toggle word wrap
3. Configure the manifest files in the cluster-manifests directory. For sample files, see the "Sample GitOps ZTP custom resources" section.

4. Disconnected clusters: If you did not define mirror configuration in the install-config.yaml file before generating the ZTP manifests, perform the following steps:

   1. Navigate to the mirror directory by running the following command:

      $ cd ../mirror

      Copy to Clipboard Toggle word wrap
   2. Configure the manifest files in the mirror directory.

Procedure

1. Generate ZTP cluster manifests by running the following command:

   $ openshift-install agent create cluster-manifests --dir <installation_directory>

   Copy to Clipboard Toggle word wrap
   Important

   If you have created the install-config.yaml and agent-config.yaml files, those files are deleted and replaced by the cluster manifests generated through this command.

   Any configurations made to the install-config.yaml and agent-config.yaml files are imported to the ZTP cluster manifests when you run the openshift-install agent create cluster-manifests command.

   Note

   If you have already generated ZTP manifests, skip this step.

2. Navigate to the cluster-manifests directory by running the following command:

   $ cd <installation_directory>/cluster-manifests

   Copy to Clipboard Toggle word wrap

3. Add the following section to the agent-cluster-install.yaml file:

   diskEncryption:
       enableOn: all 

   1

   
       mode: tang 

   2

   
       tangServers: "server1": "http://tang-server-1.example.com:7500" 

   3

   Copy to Clipboard Toggle word wrap

   1

   Specify which nodes to enable disk encryption on. Valid values are none, all, masters, and workers.

   2

   Specify which disk encryption mode to use. Valid values are tpmv2 and tang.

   3

   Optional: If you are using Tang, specify the Tang servers.

Procedure

1. Create the agent image by running the following command:

   $ openshift-install --dir <install_directory> agent create image

   Copy to Clipboard Toggle word wrap
   Note

   Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with a default /etc/multipath.conf configuration.

2. Boot the agent.x86_64.iso, agent.aarch64.iso, or agent.s390x.iso image on the bare metal machines.

Procedure

1. Boot your RHEL KVM machine.

2. To deploy the virtual server, run the virt-install command with the following parameters:

   ISO boot

   $ virt-install
       --name <vm_name> \
       --autostart \
       --memory=<memory> \
       --cpu host \
       --vcpus=<vcpus> \
       --cdrom \<path_to_image>/<agent_iso_image> \ 

   1

   
       --disk pool=default,size=<disk_pool_size> \
       --network network:default,mac=<mac_address> \
       --graphics none \
       --noautoconsole \
       --os-variant rhel9.0 \
       --wait=-1

   Copy to Clipboard Toggle word wrap

   1

   For the --cdrom parameter, specify the location of the ISO image on the local server, for example, <path_to_image>/home/<image>.iso.

3. Optional: Enable FIPS mode.

   To enable FIPS mode on IBM Z® clusters with RHEL KVM you must use PXE boot instead and run the virt-install command with the following parameters:

   PXE boot

   $ virt-install \
      --name <vm_name> \
      --autostart \
      --ram=16384 \
      --cpu host \
      --vcpus=8 \
      --location <path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img \

   1

   
      --disk <qcow_image_path> \
      --network network:macvtap ,mac=<mac_address> \
      --graphics none \
      --noautoconsole \
      --wait=-1 \
      --extra-args "rd.neednet=1 nameserver=<nameserver>" \
      --extra-args "ip=<IP>::<nameserver>::<hostname>:enc1:none" \
      --extra-args "coreos.live.rootfs_url=http://<http_server>:8080/agent.s390x-rootfs.img" \
      --extra-args "random.trust_cpu=on rd.luks.options=discard" \
      --extra-args "ignition.firstboot ignition.platform.id=metal" \
      --extra-args "console=tty1 console=ttyS1,115200n8" \
      --extra-args "coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" \
      --extra-args "fips=1" \

   2

   
      --osinfo detect=on,require=off

   Copy to Clipboard Toggle word wrap

   1

   For the --location parameter, specify the location of the kernel/initrd on the HTTP or HTTPS server.

   2

   To enable FIPS mode, specify fips=1. This entry is required in addition to setting the fips parameter to true in the install-config.yaml file.

   Note

   Currently, only PXE boot is supported to enable FIPS mode on IBM Z®.

Procedure

1. Wait for the agent console application to check whether or not the configured release image can be pulled from a registry.

2. If the agent console application states that the installer connectivity checks have passed, wait for the prompt to time out to continue with the installation.

   Note

   You can still choose to view or change network configuration settings even if the connectivity checks have passed.

   However, if you choose to interact with the agent console application rather than letting it time out, you must manually quit the TUI to proceed with the installation.

3. If the agent console application checks have failed, which is indicated by a red icon beside the Release image URL pull check, use the following steps to reconfigure the host’s network settings:

   1. Read the Check Errors section of the TUI. This section displays error messages specific to the failed checks.

      

      View larger image

      
   2. Select Configure network to launch the NetworkManager TUI.
   3. Select Edit a connection and select the connection you want to reconfigure.
   4. Edit the configuration and select OK to save your changes.
   5. Select Back to return to the main screen of the NetworkManager TUI.
   6. Select Activate a Connection.
   7. Select the reconfigured network to deactivate it.
   8. Select the reconfigured network again to reactivate it.
   9. Select Back and then select Quit to return to the agent console application.
   10. Wait at least five seconds for the continuous network checks to restart using the new network configuration.
   11. If the Release image URL pull check succeeds and displays a green icon beside the URL, select Quit to exit the agent console application and continue with the installation.

Procedure

1. Optional: To know when the bootstrap host (rendezvous host) reboots, run the following command:

   $ ./openshift-install --dir <install_directory> agent wait-for bootstrap-complete \

   1

   
       --log-level=info 

   2

   Copy to Clipboard Toggle word wrap

   1

   For <install_directory>, specify the path to the directory where the agent ISO was generated.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Example output

   ...................................................................
   ...................................................................
   INFO Bootstrap configMap status is complete
   INFO cluster bootstrap is complete

   Copy to Clipboard Toggle word wrap

   The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

2. To track the progress and verify successful installation, run the following command:

   $ openshift-install --dir <install_directory> agent wait-for install-complete 

   1

   Copy to Clipboard Toggle word wrap

   1

   For <install_directory> directory, specify the path to the directory where the agent ISO was generated.

   Example output

   ...................................................................
   ...................................................................
   INFO Cluster is installed
   INFO Install complete!
   INFO To access the cluster as the system:admin user when using 'oc', run
   INFO     export KUBECONFIG=/home/core/installer/auth/kubeconfig
   INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sno-cluster.test.example.com

   Copy to Clipboard Toggle word wrap