Installation configuration

Procedure

1. Navigate to the Butane image download page at https://mirror.openshift.com/pub/openshift-v4/clients/butane/.

2. Get the butane binary:

   1. For the newest version of Butane, save the latest butane image to your current directory:

      $ curl https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/butane --output butane

   2. Optional: For a specific type of architecture you are installing Butane on, such as aarch64 or ppc64le, indicate the appropriate URL. For example:

      $ curl https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/butane-aarch64 --output butane

3. Make the downloaded binary file executable:

   $ chmod +x butane

4. Move the butane binary file to a directory on your PATH.

   To check your PATH, open a terminal and execute the following command:

   $ echo $PATH

Procedure

1. Create a Butane config file. The following example creates a file named 99-worker-custom.bu that configures the system console to show kernel debug messages and specifies custom settings for the chrony time service:

   variant: openshift
   version: 4.16.0
   metadata:
     name: 99-worker-custom
     labels:
       machineconfiguration.openshift.io/role: worker
   openshift:
     kernel_arguments:
       - loglevel=7
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             pool 0.rhel.pool.ntp.org iburst
             driftfile /var/lib/chrony/drift
             makestep 1.0 3
             rtcsync
             logdir /var/log/chrony

   Note

   The 99-worker-custom.bu file is set to create a machine config for worker nodes. To deploy on control plane nodes, change the role from worker to master. To do both, you could repeat the whole procedure using different file names for the two types of deployments.

2. Create a MachineConfig object by giving Butane the file that you created in the previous step:

   $ butane 99-worker-custom.bu -o ./99-worker-custom.yaml

   A MachineConfig object YAML file is created for you to finish configuring your machines.

3. Save the Butane config in case you need to update the MachineConfig object in the future.

4. If the cluster is not running yet, generate manifest files and add the MachineConfig object YAML file to the openshift directory. If the cluster is already running, apply the file as follows:

   $ oc create -f 99-worker-custom.yaml

Procedure

1. Register a RHEL 8 system:

   # subscription-manager register

2. Attach a subscription to the RHEL 8 system:

   # subscription-manager attach --auto

3. Install software that is required to build the software and container:

   # yum install podman make git -y

4. Clone the kmod-via-containers repository:

   1. Create a folder for the repository:

      $ mkdir kmods; cd kmods

   2. Clone the repository:

      $ git clone https://github.com/kmods-via-containers/kmods-via-containers

5. Install a KVC framework instance on your RHEL 8 build host to test the module. This adds a kmods-via-container systemd service and loads it:

   1. Change to the kmod-via-containers directory:

      $ cd kmods-via-containers/

   2. Install the KVC framework instance:

      $ sudo make install

   3. Reload the systemd manager configuration:

      $ sudo systemctl daemon-reload

6. Get the kernel module source code. The source code might be used to build a third-party module that you do not have control over, but is supplied by others. You will need content similar to the content shown in the kvc-simple-kmod example that can be cloned to your system as follows:

   $ cd .. ; git clone https://github.com/kmods-via-containers/kvc-simple-kmod

7. Edit the configuration file, simple-kmod.conf file, in this example, and change the name of the Dockerfile to Dockerfile.rhel:

   1. Change to the kvc-simple-kmod directory:

      $ cd kvc-simple-kmod

   2. Rename the Dockerfile:

      $ cat simple-kmod.conf

      Example Dockerfile

      KMOD_CONTAINER_BUILD_CONTEXT="https://github.com/kmods-via-containers/kvc-simple-kmod.git"
      KMOD_CONTAINER_BUILD_FILE=Dockerfile.rhel
      KMOD_SOFTWARE_VERSION=dd1a7d4
      KMOD_NAMES="simple-kmod simple-procfs-kmod"

8. Create an instance of kmods-via-containers@.service for your kernel module, simple-kmod in this example:

   $ sudo make install

9. Enable the kmods-via-containers@.service instance:

   $ sudo kmods-via-containers build simple-kmod $(uname -r)

10. Enable and start the systemd service:

    $ sudo systemctl enable kmods-via-containers@simple-kmod.service --now

    1. Review the service status:

       $ sudo systemctl status kmods-via-containers@simple-kmod.service

       Example output

       ● kmods-via-containers@simple-kmod.service - Kmods Via Containers - simple-kmod
          Loaded: loaded (/etc/systemd/system/kmods-via-containers@.service;
                 enabled; vendor preset: disabled)
          Active: active (exited) since Sun 2020-01-12 23:49:49 EST; 5s ago...

11. To confirm that the kernel modules are loaded, use the lsmod command to list the modules:

    $ lsmod | grep simple_

    Example output

    simple_procfs_kmod     16384  0
    simple_kmod            16384  0

12. Optional. Use other methods to check that the simple-kmod example is working:

    • Look for a "Hello world" message in the kernel ring buffer with dmesg:

      $ dmesg | grep 'Hello world'

      Example output

      [ 6420.761332] Hello world from simple_kmod.

    • Check the value of simple-procfs-kmod in /proc:

      $ sudo cat /proc/simple-procfs-kmod

      Example output

      simple-procfs-kmod number = 0

    • Run the spkut command to get more information from the module:

      $ sudo spkut 44

      Example output

      KVC: wrapper simple-kmod for 4.18.0-147.3.1.el8_1.x86_64
      Running userspace wrapper using the kernel module container...
      + podman run -i --rm --privileged
         simple-kmod-dd1a7d4:4.18.0-147.3.1.el8_1.x86_64 spkut 44
      simple-procfs-kmod number = 0
      simple-procfs-kmod number = 44

Procedure

1. If you want to use TPM v2 to encrypt your cluster, check to see if TPM v2 encryption needs to be enabled in the host firmware for each node. This is required on most Dell systems. Check the manual for your specific system.

2. If you want to use Tang to encrypt your cluster, follow these preparatory steps:

   1. Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions.

   2. Install the clevis package on a RHEL 8 machine, if it is not already installed:

      $ sudo yum install clevis

   3. On the RHEL 8 machine, run the following command to generate a thumbprint of the exchange key. Replace http://tang1.example.com:7500 with the URL of your Tang server:

      $ clevis-encrypt-tang '{"url":"http://tang1.example.com:7500"}' < /dev/null > /dev/null 1

      1

      In this example, tangd.socket is listening on port 7500 on the Tang server.

      Note

      The clevis-encrypt-tang command generates a thumbprint of the exchange key. No data passes to the encryption command during this step; /dev/null exists here as an input instead of plain text. The encrypted output is also sent to /dev/null, because it is not required for this procedure.

      Example output

      The advertisement contains the following signing keys:
      
      PLjNyRdGw03zlRoGjQYMahSZGu9 1

      1

      The thumbprint of the exchange key.

      When the Do you wish to trust these keys? [ynYN] prompt displays, type Y.

   4. Optional: For offline Tang provisioning:

      1. Obtain the advertisement from the server using the curl command. Replace http://tang2.example.com:7500 with the URL of your Tang server:

         $ curl -f http://tang2.example.com:7500/adv > adv.jws && cat adv.jws

         Expected output

         {"payload": "eyJrZXlzIjogW3siYWxnIjogIkV", "protected": "eyJhbGciOiJFUzUxMiIsImN0eSI", "signature": "ADLgk7fZdE3Yt4FyYsm0pHiau7Q"}

      2. Provide the advertisement file to Clevis for encryption:

         $ clevis-encrypt-tang '{"url":"http://tang2.example.com:7500","adv":"adv.jws"}' < /dev/null > /dev/null

   5. If the nodes are configured with static IP addressing, run coreos-installer iso customize --dest-karg-append or use the coreos-installer --append-karg option when installing RHCOS nodes to set the IP address of the installed system. Append the ip= and other arguments needed for your network.

      Important

      Some methods for configuring static IPs do not affect the initramfs after the first boot and will not work with Tang encryption. These include the coreos-installer --copy-network option, the coreos-installer iso customize --network-keyfile option, and the coreos-installer pxe customize --network-keyfile option, as well as adding ip= arguments to the kernel command line of the live ISO or PXE image during installation. Incorrect static IP configuration causes the second boot of the node to fail.

3. On your installation node, change to the directory that contains the installation program and generate the Kubernetes manifests for the cluster:

   $ ./openshift-install create manifests --dir <installation_directory> 1

   1

   Replace <installation_directory> with the path to the directory that you want to store the installation files in.

4. Create a Butane config that configures disk encryption, mirroring, or both. For example, to configure storage for compute nodes, create a $HOME/clusterconfig/worker-storage.bu file.

   Butane config example for a boot device

   variant: openshift
   version: 4.16.0
   metadata:
     name: worker-storage 1
     labels:
       machineconfiguration.openshift.io/role: worker 2
   boot_device:
     layout: x86_64 3
     luks: 4
       tpm2: true 5
       tang: 6
         - url: http://tang1.example.com:7500 7
           thumbprint: PLjNyRdGw03zlRoGjQYMahSZGu9 8
         - url: http://tang2.example.com:7500
           thumbprint: VCJsvZFjBSIHSldw78rOrq7h2ZF
           advertisement: "{"payload": "eyJrZXlzIjogW3siYWxnIjogIkV", "protected": "eyJhbGciOiJFUzUxMiIsImN0eSI", "signature": "ADLgk7fZdE3Yt4FyYsm0pHiau7Q"}" 9
       threshold: 1 10
     mirror: 11
       devices: 12
         - /dev/sda
         - /dev/sdb
   openshift:
     fips: true 13

   1 2

   For control plane configurations, replace worker with master in both of these locations.

   3

   Set this field to the instruction set architecture of the cluster nodes. Some examples include, x86_64, aarch64, or ppc64le.

   4

   Include this section if you want to encrypt the root file system. For more details, see "About disk encryption".

   5

   Include this field if you want to use a Trusted Platform Module (TPM) to encrypt the root file system.

   6

   Include this section if you want to use one or more Tang servers.

   7

   Specify the URL of a Tang server. In this example, tangd.socket is listening on port 7500 on the Tang server.

   8

   Specify the exchange key thumbprint, which was generated in a preceding step.

   9

   Optional: Specify the advertisement for your offline Tang server in valid JSON format.

   10

   Specify the minimum number of TPM v2 and Tang encryption conditions that must be met for decryption to occur. The default value is 1. For more information about this topic, see "Configuring an encryption threshold".

   11

   Include this section if you want to mirror the boot disk. For more details, see "About disk mirroring".

   12

   List all disk devices that should be included in the boot disk mirror, including the disk that RHCOS will be installed onto.

   13

   Include this directive to enable FIPS mode on your cluster.

   Important

   To enable FIPS mode for your cluster, you must run the installation program from a Red Hat Enterprise Linux (RHEL) computer configured to operate in FIPS mode. For more information about configuring FIPS mode on RHEL, see Installing the system in FIPS mode. If you are configuring nodes to use both disk encryption and mirroring, both features must be configured in the same Butane configuration file. If you are configuring disk encryption on a node with FIPS mode enabled, you must include the fips directive in the same Butane configuration file, even if FIPS mode is also enabled in a separate manifest.

5. Create a control plane or compute node manifest from the corresponding Butane configuration file and save it to the <installation_directory>/openshift directory. For example, to create a manifest for the compute nodes, run the following command:

   $ butane $HOME/clusterconfig/worker-storage.bu -o <installation_directory>/openshift/99-worker-storage.yaml

   Repeat this step for each node type that requires disk encryption or mirroring.

6. Save the Butane configuration file in case you need to update the manifests in the future.

7. Continue with the remainder of the OpenShift Container Platform installation.

   Tip

   You can monitor the console log on the RHCOS nodes during installation for error messages relating to disk encryption or mirroring.

   Important

   If you configure additional data partitions, they will not be encrypted unless encryption is explicitly requested.

1. From the installation host, access a cluster node by using a debug pod:

   1. Start a debug pod for the node, for example:

      $ oc debug node/compute-1

   2. Set /host as the root directory within the debug shell. The debug pod mounts the root file system of the node in /host within the pod. By changing the root directory to /host, you can run binaries contained in the executable paths on the node:

      # chroot /host

      Note

      OpenShift Container Platform cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

2. If you configured boot disk encryption, verify if it is enabled:

   1. From the debug shell, review the status of the root mapping on the node:

      # cryptsetup status root

      Example output

      /dev/mapper/root is active and is in use.
        type:    LUKS2 1
        cipher:  aes-xts-plain64 2
        keysize: 512 bits
        key location: keyring
        device:  /dev/sda4 3
        sector size:  512
        offset:  32768 sectors
        size:    15683456 sectors
        mode:    read/write

      1

      The encryption format. When the TPM v2 or Tang encryption modes are enabled, the RHCOS boot disks are encrypted using the LUKS2 format.

      2

      The encryption algorithm used to encrypt the LUKS2 volume. The aes-cbc-essiv:sha256 cipher is used if FIPS mode is enabled.

      3

      The device that contains the encrypted LUKS2 volume. If mirroring is enabled, the value will represent a software mirror device, for example /dev/md126.

   2. List the Clevis plugins that are bound to the encrypted device:

      # clevis luks list -d /dev/sda4 1

      1

      Specify the device that is listed in the device field in the output of the preceding step.

      Example output

      1: sss '{"t":1,"pins":{"tang":[{"url":"http://tang.example.com:7500"}]}}' 1

      1

      In the example output, the Tang plugin is used by the Shamir’s Secret Sharing (SSS) Clevis plugin for the /dev/sda4 device.

3. If you configured mirroring, verify if it is enabled:

   1. From the debug shell, list the software RAID devices on the node:

      # cat /proc/mdstat

      Example output

      Personalities : [raid1]
      md126 : active raid1 sdb3[1] sda3[0] 1
      	  393152 blocks super 1.0 [2/2] [UU]
      
      md127 : active raid1 sda4[0] sdb4[1] 2
      	  51869632 blocks super 1.2 [2/2] [UU]
      
      unused devices: <none>

      1

      The /dev/md126 software RAID mirror device uses the /dev/sda3 and /dev/sdb3 disk devices on the cluster node.

      2

      The /dev/md127 software RAID mirror device uses the /dev/sda4 and /dev/sdb4 disk devices on the cluster node.

   2. Review the details of each of the software RAID devices listed in the output of the preceding command. The following example lists the details of the /dev/md126 device:

      # mdadm --detail /dev/md126

      Example output

      /dev/md126:
                 Version : 1.0
           Creation Time : Wed Jul  7 11:07:36 2021
              Raid Level : raid1 1
              Array Size : 393152 (383.94 MiB 402.59 MB)
           Used Dev Size : 393152 (383.94 MiB 402.59 MB)
            Raid Devices : 2
           Total Devices : 2
             Persistence : Superblock is persistent
      
             Update Time : Wed Jul  7 11:18:24 2021
                   State : clean 2
          Active Devices : 2 3
         Working Devices : 2 4
          Failed Devices : 0 5
           Spare Devices : 0
      
      Consistency Policy : resync
      
                    Name : any:md-boot 6
                    UUID : ccfa3801:c520e0b5:2bee2755:69043055
                  Events : 19
      
          Number   Major   Minor   RaidDevice State
             0     252        3        0      active sync   /dev/sda3 7
             1     252       19        1      active sync   /dev/sdb3 8

      1

      Specifies the RAID level of the device. raid1 indicates RAID 1 disk mirroring.

      2

      Specifies the state of the RAID device.

      3 4

      States the number of underlying disk devices that are active and working.

      5

      States the number of underlying disk devices that are in a failed state.

      6

      The name of the software RAID device.

      7 8

      Provides information about the underlying disk devices used by the software RAID device.

   3. List the file systems mounted on the software RAID devices:

      # mount | grep /dev/md

      Example output

      /dev/md127 on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /etc type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /usr type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /sysroot type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/containers/storage/overlay type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/kubelet/pods/e5054ed5-f882-4d14-b599-99c050d4e0c0/volume-subpaths/etc/tuned/1 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/kubelet/pods/e5054ed5-f882-4d14-b599-99c050d4e0c0/volume-subpaths/etc/tuned/2 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/kubelet/pods/e5054ed5-f882-4d14-b599-99c050d4e0c0/volume-subpaths/etc/tuned/3 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/kubelet/pods/e5054ed5-f882-4d14-b599-99c050d4e0c0/volume-subpaths/etc/tuned/4 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md127 on /var/lib/kubelet/pods/e5054ed5-f882-4d14-b599-99c050d4e0c0/volume-subpaths/etc/tuned/5 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
      /dev/md126 on /boot type ext4 (rw,relatime,seclabel)

      In the example output, the /boot file system is mounted on the /dev/md126 software RAID device and the root file system is mounted on /dev/md127.

4. Repeat the verification steps for each OpenShift Container Platform node type.

Procedure

1. Create a Butane config that configures a data volume by using software RAID.

   • To configure a data volume with RAID 1 on the same disks that are used for a mirrored boot disk, create a $HOME/clusterconfig/raid1-storage.bu file, for example:

     RAID 1 on mirrored boot disk

     variant: openshift
     version: 4.16.0
     metadata:
       name: raid1-storage
       labels:
         machineconfiguration.openshift.io/role: worker
     boot_device:
       mirror:
         devices:
           - /dev/disk/by-id/scsi-3600508b400105e210000900000490000
           - /dev/disk/by-id/scsi-SSEAGATE_ST373453LW_3HW1RHM6
     storage:
       disks:
         - device: /dev/disk/by-id/scsi-3600508b400105e210000900000490000
           partitions:
             - label: root-1
               size_mib: 25000 1
             - label: var-1
         - device: /dev/disk/by-id/scsi-SSEAGATE_ST373453LW_3HW1RHM6
           partitions:
             - label: root-2
               size_mib: 25000 2
             - label: var-2
       raid:
         - name: md-var
           level: raid1
           devices:
             - /dev/disk/by-partlabel/var-1
             - /dev/disk/by-partlabel/var-2
       filesystems:
         - device: /dev/md/md-var
           path: /var
           format: xfs
           wipe_filesystem: true
           with_mount_unit: true

     1 2

     When adding a data partition to the boot disk, a minimum value of 25000 mebibytes is recommended. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition.

   • To configure a data volume with RAID 1 on secondary disks, create a $HOME/clusterconfig/raid1-alt-storage.bu file, for example:

     RAID 1 on secondary disks

     variant: openshift
     version: 4.16.0
     metadata:
       name: raid1-alt-storage
       labels:
         machineconfiguration.openshift.io/role: worker
     storage:
       disks:
         - device: /dev/sdc
           wipe_table: true
           partitions:
             - label: data-1
         - device: /dev/sdd
           wipe_table: true
           partitions:
             - label: data-2
       raid:
         - name: md-var-lib-containers
           level: raid1
           devices:
             - /dev/disk/by-partlabel/data-1
             - /dev/disk/by-partlabel/data-2
       filesystems:
         - device: /dev/md/md-var-lib-containers
           path: /var/lib/containers
           format: xfs
           wipe_filesystem: true
           with_mount_unit: true

2. Create a RAID manifest from the Butane config you created in the previous step and save it to the <installation_directory>/openshift directory. For example, to create a manifest for the compute nodes, run the following command:

   $ butane $HOME/clusterconfig/<butane_config>.bu -o <installation_directory>/openshift/<manifest_name>.yaml 1

   1

   Replace <butane_config> and <manifest_name> with the file names from the previous step. For example, raid1-alt-storage.bu and raid1-alt-storage.yaml for secondary disks.

3. Save the Butane config in case you need to update the manifest in the future.
4. Continue with the remainder of the OpenShift Container Platform installation.

Procedure

1. Create the Intel® Matrix Storage Manager (IMSM) RAID container by running the following command:

   $ mdadm -CR /dev/md/imsm0 -e \
     imsm -n2 /dev/nvme0n1 /dev/nvme1n1 1

   1

   The RAID device names. In this example, there are two devices listed. If you provide more than two device names, you must adjust the -n flag. For example, listing three devices would use the flag -n3.

2. Create the RAID1 storage inside the container:

   1. Create a dummy RAID0 volume in front of the real RAID1 volume by running the following command:

      $ mdadm -CR /dev/md/dummy -l0 -n2 /dev/imsm0 -z10M --assume-clean

   2. Create the real RAID1 array by running the following command:

      $ mdadm -CR /dev/md/coreos -l1 -n2 /dev/imsm0

   3. Stop both RAID0 and RAID1 member arrays and delete the dummy RAID0 array with the following commands:

      $ mdadm -S /dev/md/dummy \
        mdadm -S /dev/md/coreos \
        mdadm --kill-subarray=0 /dev/md/imsm0

   4. Restart the RAID1 arrays by running the following command:

      $ mdadm -A /dev/md/coreos /dev/md/imsm0

3. Install RHCOS on the RAID1 device:

   1. Get the UUID of the IMSM container by running the following command:

      $ mdadm --detail --export /dev/md/imsm0

   2. Install RHCOS and include the rd.md.uuid kernel argument by running the following command:

      $ coreos-installer install /dev/md/coreos \
        --append-karg rd.md.uuid=<md_UUID> 1
        ...

      1

      The UUID of the IMSM container.

      Include any additional coreos-installer arguments you need to install RHCOS.