Chapter 8. Deploying installer-provisioned clusters on bare metal

1. Two unique virtual IP addresses.

   • One virtual IP address for the API endpoint.
   • One virtual IP address for the wildcard ingress endpoint.
2. One IP address for the provisioner node.
3. One IP address for each control plane (master) node.
4. One IP address for each worker node, if applicable.

1. Configure install-config.yaml. Change the appropriate variables to match the environment, including pullSecret and sshKey.

   apiVersion: v1
   baseDomain: <domain>
   metadata:
     name: <cluster-name>
   networking:
     machineCIDR: <public-cidr>
     networkType: OVNKubernetes
   compute:
   - name: worker
     replicas: 2 

   1

   
   controlPlane:
     name: master
     replicas: 3
     platform:
       baremetal: {}
   platform:
     baremetal:
       apiVIP: <api-ip>
       ingressVIP: <wildcard-ip>
       provisioningNetworkCIDR: <CIDR>
       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 

   2

   
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           rootDeviceHints:
            deviceName: "/dev/disk/by-id/<disk_id>" 

   3

   
         - name: <openshift-master-1>
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 

   4

   
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           rootDeviceHints:
            deviceName: "/dev/disk/by-id/<disk_id>" 

   5

   
         - name: <openshift-master-2>
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 

   6

   
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           rootDeviceHints:
            deviceName: "/dev/disk/by-id/<disk_id>" 

   7

   
         - name: <openshift-worker-0>
           role: worker
           bmc:
             address: ipmi://<out-of-band-ip> 

   8

   
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
         - name: <openshift-worker-1>
           role: worker
           bmc:
             address: ipmi://<out-of-band-ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           rootDeviceHints:
            deviceName: "/dev/disk/by-id/<disk_id>" 

   9

   
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_pub_key>'

   Copy to Clipboard Toggle word wrap

   1

   Scale the worker machines based on the number of worker nodes that are part of the OpenShift Container Platform cluster.

   2 4 6 8

   See the BMC addressing sections for more options.

   3 5 7 9

   Set the path to the installation disk drive, for example, /dev/disk/by-id/wwn-0x64cd98f04fde100024684cf3034da5c2.

2. Create a directory to store cluster configs.

   $ mkdir ~/clusterconfigs
   $ cp install-config.yaml ~/clusterconfigs

   Copy to Clipboard Toggle word wrap

3. Ensure all bare metal nodes are powered off prior to installing the OpenShift Container Platform cluster.

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

   Copy to Clipboard Toggle word wrap

4. Remove old bootstrap resources if any are left over from a previous deployment attempt.

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

   Copy to Clipboard Toggle word wrap

1. Create the OpenShift Container Platform manifests.

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

   Copy to Clipboard Toggle word wrap

   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
   WARNING Discarding the OpenShift Manifest that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 

   1

   
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all worker nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   Copy to Clipboard Toggle word wrap

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

   Copy to Clipboard Toggle word wrap

3. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the worker nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.8.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 

   1

   
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   Copy to Clipboard Toggle word wrap

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

4. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Change to the directory storing the install-config.yaml file:

   $ cd ~/clusterconfigs

   Copy to Clipboard Toggle word wrap

2. Switch to the manifests subdirectory:

   $ cd manifests

   Copy to Clipboard Toggle word wrap

3. Create a file named cluster-network-avoid-workers-99-config.yaml:

   $ touch cluster-network-avoid-workers-99-config.yaml

   Copy to Clipboard Toggle word wrap

4. Open the cluster-network-avoid-workers-99-config.yaml file in an editor and enter a custom resource (CR) that describes the Operator configuration:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: 50-worker-fix-ipi-rwn
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/kubernetes/manifests/keepalived.yaml
             mode: 0644
             contents:
               source: data:,

   Copy to Clipboard Toggle word wrap

   This manifest places the ingressVIP virtual IP address on the control plane nodes. Additionally, this manifest deploys the following processes on the control plane nodes only:

   • openshift-ingress-operator
   • keepalived
5. Save the cluster-network-avoid-workers-99-config.yaml file.

6. Create a manifests/cluster-ingress-default-ingresscontroller.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/master: ""

   Copy to Clipboard Toggle word wrap
7. Consider backing up the manifests directory. The installer deletes the manifests/ directory when creating the cluster.

8. Modify the cluster-scheduler-02-config.yml manifest to make the control plane nodes schedulable by setting the mastersSchedulable field to true. Control plane nodes are not schedulable by default. For example:

   $ sed -i "s;mastersSchedulable: false;mastersSchedulable: true;g" clusterconfigs/manifests/cluster-scheduler-02-config.yml

   Copy to Clipboard Toggle word wrap
   Note

   If control plane nodes are not schedulable after completing this procedure, deploying the cluster will fail.

Procedure

1. Open the firewall port on the registry node.

   $ sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent
   $ sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent
   $ sudo firewall-cmd --reload

   Copy to Clipboard Toggle word wrap

2. Install the required packages for the registry node.

   $ sudo yum -y install python3 podman httpd httpd-tools jq

   Copy to Clipboard Toggle word wrap

3. Create the directory structure where the repository information will be held.

   $ sudo mkdir -p /opt/registry/{auth,certs,data}

   Copy to Clipboard Toggle word wrap

Procedure

1. Adjust the certificate information as appropriate.

   $ host_fqdn=$( hostname --long )
   $ cert_c="<Country Name>"   # Country Name (C, 2 letter code)
   $ cert_s="<State>"          # Certificate State (S)
   $ cert_l="<Locality>"       # Certificate Locality (L)
   $ cert_o="<Organization>"   # Certificate Organization (O)
   $ cert_ou="<Org Unit>"      # Certificate Organizational Unit (OU)
   $ cert_cn="${host_fqdn}"    # Certificate Common Name (CN)
   
   $ openssl req \
       -newkey rsa:4096 \
       -nodes \
       -sha256 \
       -keyout /opt/registry/certs/domain.key \
       -x509 \
       -days 365 \
       -out /opt/registry/certs/domain.crt \
       -addext "subjectAltName = DNS:${host_fqdn}" \
       -subj "/C=${cert_c}/ST=${cert_s}/L=${cert_l}/O=${cert_o}/OU=${cert_ou}/CN=${cert_cn}"

   Copy to Clipboard Toggle word wrap
   Note

   When replacing <Country Name>, ensure that it only contains two letters. For example, US.

2. Update the registry node’s ca-trust with the new certificate.

   $ sudo cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
   $ sudo update-ca-trust extract

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an htpasswd file in /opt/registry/auth for the container to use.

   $ htpasswd -bBc /opt/registry/auth/htpasswd <user> <passwd>

   Copy to Clipboard Toggle word wrap

   Replace <user> with the user name and <passwd> with the password.

2. Create and start the registry container.

   $ podman create \
     --name ocpdiscon-registry \
     -p 5000:5000 \
     -e "REGISTRY_AUTH=htpasswd" \
     -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \
     -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
     -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
     -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
     -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
     -e "REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true" \
     -v /opt/registry/data:/var/lib/registry:z \
     -v /opt/registry/auth:/auth:z \
     -v /opt/registry/certs:/certs:z \
     docker.io/library/registry:2

   Copy to Clipboard Toggle word wrap

   $ podman start ocpdiscon-registry

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the pull-secret.txt file.

   $ scp kni@provisioner:/home/kni/pull-secret.txt pull-secret.txt

   Copy to Clipboard Toggle word wrap

2. Update the host_fqdn environment variable with the fully qualified domain name of the registry node.

   $ host_fqdn=$( hostname --long )

   Copy to Clipboard Toggle word wrap

3. Update the b64auth environment variable with the base64 encoding of the http credentials used to create the htpasswd file.

   $ b64auth=$( echo -n '<username>:<passwd>' | openssl base64 )

   Copy to Clipboard Toggle word wrap

   Replace <username> with the user name and <passwd> with the password.

4. Set the AUTHSTRING environment variable to use the base64 authorization string. The $USER variable is an environment variable containing the name of the current user.

   $ AUTHSTRING="{\"$host_fqdn:5000\": {\"auth\": \"$b64auth\",\"email\": \"$USER@redhat.com\"}}"

   Copy to Clipboard Toggle word wrap

5. Update the pull-secret.txt file.

   $ jq ".auths += $AUTHSTRING" < pull-secret.txt > pull-secret-update.txt

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the oc binary from the provisioner node to the registry node.

   $ sudo scp kni@provisioner:/usr/local/bin/oc /usr/local/bin

   Copy to Clipboard Toggle word wrap

2. Set the required environment variables.

   1. Set the release version:

      $ VERSION=<release_version>

      Copy to Clipboard Toggle word wrap

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.8.

   2. Set the local registry name and host port:

      $ LOCAL_REG='<local_registry_host_name>:<local_registry_host_port>'

      Copy to Clipboard Toggle word wrap

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Set the local repository name:

      $ LOCAL_REPO='<local_repository_name>'

      Copy to Clipboard Toggle word wrap

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

3. Mirror the remote install images to the local repository.

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.txt \
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Copy to Clipboard Toggle word wrap

Procedure

1. Add the disconnected registry node’s certificate to the install-config.yaml file. The certificate should follow the "additionalTrustBundle: |" line and be properly indented, usually by two spaces.

   $ echo "additionalTrustBundle: |" >> install-config.yaml
   $ sed -e 's/^/  /' /opt/registry/certs/domain.crt >> install-config.yaml

   Copy to Clipboard Toggle word wrap

2. Add the mirror information for the registry to the install-config.yaml file.

   $ echo "imageContentSources:" >> install-config.yaml
   $ echo "- mirrors:" >> install-config.yaml
   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
   $ echo "  source: quay.io/openshift-release-dev/ocp-release" >> install-config.yaml
   $ echo "- mirrors:" >> install-config.yaml
   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
   $ echo "  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev" >> install-config.yaml

   Copy to Clipboard Toggle word wrap
   Note

   Replace registry.example.com with the registry’s fully qualified domain name.