Deploying installer-provisioned clusters on bare metal

1. Two unique virtual IP addresses.

   • One virtual IP address for the API endpoint.
   • One virtual IP address for the wildcard ingress endpoint.
2. One IP address for the provisioner node.
3. One IP address for each control plane node.
4. One IP address for each worker node, if applicable.

1. Configure install-config.yaml. Change the appropriate variables to match the environment, including pullSecret and sshKey:

   apiVersion: v1
   baseDomain: <domain>
   metadata:
     name: <cluster_name>
   networking:
     machineNetwork:
     - cidr: <public_cidr>
     networkType: OVNKubernetes
   compute:
   - name: worker
     replicas: 2 1
   controlPlane:
     name: master
     replicas: 3
     platform:
       baremetal: {}
   platform:
     baremetal:
       apiVIPs:
         - <api_ip>
       ingressVIPs:
         - <wildcard_ip>
       provisioningNetworkCIDR: <CIDR>
       bootstrapExternalStaticIP: <bootstrap_static_ip_address> 2
       bootstrapExternalStaticGateway: <bootstrap_static_gateway> 3
       bootstrapExternalStaticDNS: <bootstrap_static_dns> 4
       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: ipmi://<out_of_band_ip> 5
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>" 6
         - name: <openshift_master_1>
           role: master
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
         - name: <openshift_master_2>
           role: master
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
         - name: <openshift_worker_0>
           role: worker
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
         - name: <openshift_worker_1>
           role: worker
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_pub_key>'

   1

   Scale the worker machines based on the number of worker nodes that are part of the OpenShift Container Platform cluster. Valid options for the replicas value are 0 and integers greater than or equal to 2. Set the number of replicas to 0 to deploy a three-node cluster, which contains only three control plane machines. A three-node cluster is a smaller, more resource-efficient cluster that can be used for testing, development, and production. You cannot install the cluster with only one worker.

   2

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticIP configuration setting to specify the static IP address of the bootstrap VM when there is no DHCP server on the bare-metal network.

   3

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticGateway configuration setting to specify the gateway IP address for the bootstrap VM when there is no DHCP server on the bare-metal network.

   4

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticDNS configuration setting to specify the DNS address for the bootstrap VM when there is no DHCP server on the bare-metal network.

   5

   See the BMC addressing sections for more options.

   6

   To set the path to the installation disk drive, enter the kernel name of the disk. For example, /dev/sda.

   Important

   Because the disk discovery order is not guaranteed, the kernel name of the disk can change across booting options for machines with multiple disks. For example, /dev/sda becomes /dev/sdb and vice versa. To avoid this issue, you must use persistent disk attributes, such as the disk World Wide Name (WWN) or /dev/disk/by-path/. It is recommended to use the /dev/disk/by-path/<device_path> link to the storage location. To use the disk WWN, replace the deviceName parameter with the wwnWithExtension parameter. Depending on the parameter that you use, enter either of the following values:

   • The disk name. For example, /dev/sda, or /dev/disk/by-path/.
   • The disk WWN. For example, "0x64cd98f04fde100024684cf3034da5c2". Ensure that you enter the disk WWN value within quotes so that it is used as a string value and not a hexadecimal value.

   Failure to meet these requirements for the rootDeviceHints parameter might result in the following error:

   ironic-inspector inspection failed: No disks satisfied root device hints

   Note

   Before OpenShift Container Platform 4.12, the cluster installation program only accepted an IPv4 address or an IPv6 address for the apiVIP and ingressVIP configuration settings. In OpenShift Container Platform 4.12 and later, these configuration settings are deprecated. Instead, use a list format in the apiVIPs and ingressVIPs configuration settings to specify IPv4 addresses, IPv6 addresses, or both IP address formats.

2. Create a directory to store the cluster configuration:

   $ mkdir ~/clusterconfigs

3. Copy the install-config.yaml file to the new directory:

   $ cp install-config.yaml ~/clusterconfigs

4. Ensure all bare metal nodes are powered off prior to installing the OpenShift Container Platform cluster:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

5. Remove old bootstrap resources if any are left over from a previous deployment attempt:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

Procedure

1. Optional: Consider testing the NMState syntax with nmstatectl gc before including it in the install-config.yaml file, because the installer will not check the NMState YAML syntax.

   Note

   Errors in the YAML syntax might result in a failure to apply the network configuration. Additionally, maintaining the validated YAML syntax is useful when applying changes using Kubernetes NMState after deployment or when expanding the cluster.

   1. Create an NMState YAML file:

      interfaces:
      - name: <nic1_name> 1
        type: ethernet
        state: up
        ipv4:
          address:
          - ip: <ip_address> 2
            prefix-length: 24
          enabled: true
      dns-resolver:
        config:
          server:
          - <dns_ip_address> 3
      routes:
        config:
        - destination: 0.0.0.0/0
          next-hop-address: <next_hop_ip_address> 4
          next-hop-interface: <next_hop_nic1_name> 5

      1 2 3 4 5

      Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

   2. Test the configuration file by running the following command:

      $ nmstatectl gc <nmstate_yaml_file>

      Replace <nmstate_yaml_file> with the configuration file name.

2. Use the networkConfig configuration setting by adding the NMState configuration to hosts within the install-config.yaml file:

       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: null
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           rootDeviceHints:
             deviceName: "/dev/sda"
           networkConfig: 1
             interfaces:
             - name: <nic1_name> 2
               type: ethernet
               state: up
               ipv4:
                 address:
                 - ip: <ip_address> 3
                   prefix-length: 24
                 enabled: true
             dns-resolver:
               config:
                 server:
                 - <dns_ip_address> 4
             routes:
               config:
               - destination: 0.0.0.0/0
                 next-hop-address: <next_hop_ip_address> 5
                 next-hop-interface: <next_hop_nic1_name> 6

   1

   Add the NMState YAML syntax to configure the host interfaces.

   2 3 4 5 6

   Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

   Important

   After deploying the cluster, you cannot modify the networkConfig configuration setting of install-config.yaml file to make changes to the host network interface. Use the Kubernetes NMState Operator to make changes to the host network interface after deployment.

Procedure

1. Add the subnets to the machineNetwork in the install-config.yaml file when using static IP addresses:

   networking:
     machineNetwork:
     - cidr: 10.0.0.0/24
     - cidr: 192.168.0.0/24
     networkType: OVNKubernetes

2. Add the gateway and DNS configuration to the networkConfig parameter of each edge compute node using NMState syntax when using a static IP address or advanced networking such as bonds:

   networkConfig:
     interfaces:
     - name: <interface_name> 1
       type: ethernet
       state: up
       ipv4:
         enabled: true
         dhcp: false
         address:
         - ip: <node_ip> 2
           prefix-length: 24
         gateway: <gateway_ip> 3
     dns-resolver:
       config:
         server:
         - <dns_ip> 4

   1

   Replace <interface_name> with the interface name.

   2

   Replace <node_ip> with the IP address of the node.

   3

   Replace <gateway_ip> with the IP address of the gateway.

   4

   Replace <dns_ip> with the IP address of the DNS server.

Procedure

1. Optional: Consider testing the NMState YAML syntax with the nmstatectl gc command before including it in the install-config.yaml file because the installation program will not check the NMState YAML syntax.

   1. Create an NMState YAML file:

      interfaces:
      - name: eth0
        ipv6:
          addr-gen-mode: <address_mode> 1

      1

      Replace <address_mode> with the type of address generation mode required for IPv6 addresses in the cluster. Valid values are eui64, stable-privacy, or random.

   2. Test the configuration file by running the following command:

      $ nmstatectl gc <nmstate_yaml_file> 1

      1

      Replace <nmstate_yaml_file> with the name of the test configuration file.

2. Add the NMState configuration to the hosts.networkConfig section within the install-config.yaml file:

       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: null
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           rootDeviceHints:
             deviceName: "/dev/sda"
           networkConfig:
             interfaces:
             - name: eth0
               ipv6:
                 addr-gen-mode: <address_mode> 1
   ...

   1

   Replace <address_mode> with the type of address generation mode required for IPv6 addresses in the cluster. Valid values are eui64, stable-privacy, or random.

Procedure

1. Add the NMState configuration to the networkConfig field to hosts within the install-config.yaml file:

       hosts:
         - name: worker-0
           role: worker
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: false
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           networkConfig: 1
             interfaces: 2
              - name: eno1 3
                type: ethernet 4
                state: up
                mac-address: 0c:42:a1:55:f3:06
                ipv4:
                  enabled: true
                  dhcp: false 5
                ethernet:
                  sr-iov:
                    total-vfs: 2 6
                ipv6:
                  enabled: false
                  dhcp: false
              - name: sriov:eno1:0
                type: ethernet
                state: up 7
                ipv4:
                  enabled: false 8
                ipv6:
                  enabled: false
              - name: sriov:eno1:1
                type: ethernet
                state: down
              - name: eno2
                type: ethernet
                state: up
                mac-address: 0c:42:a1:55:f3:07
                ipv4:
                  enabled: true
                ethernet:
                  sr-iov:
                    total-vfs: 2
                ipv6:
                  enabled: false
              - name: sriov:eno2:0
                type: ethernet
                state: up
                ipv4:
                  enabled: false
                ipv6:
                  enabled: false
              - name: sriov:eno2:1
                type: ethernet
                state: down
              - name: bond0
                type: bond
                state: up
                min-tx-rate: 100 9
                max-tx-rate: 200 10
                link-aggregation:
                  mode: active-backup 11
                  options:
                    primary: sriov:eno1:0 12
                  port:
                    - sriov:eno1:0
                    - sriov:eno2:0
                ipv4:
                  address:
                    - ip: 10.19.16.57 13
                      prefix-length: 23
                  dhcp: false
                  enabled: true
                ipv6:
                  enabled: false
             dns-resolver:
               config:
                 server:
                   - 10.11.5.160
                   - 10.2.70.215
             routes:
               config:
                 - destination: 0.0.0.0/0
                   next-hop-address: 10.19.17.254
                   next-hop-interface: bond0 14
                   table-id: 254

   1

   The networkConfig field has information about the network configuration of the host, with subfields including interfaces, dns-resolver, and routes.

   2

   The interfaces field is an array of network interfaces defined for the host.

   3

   The name of the interface.

   4

   The type of interface. This example creates a ethernet interface.

   5

   Set this to `false to disable DHCP for the physical function (PF) if it is not strictly required.

   6

   Set to the number of SR-IOV virtual functions (VFs) to instantiate.

   7

   Set this to up.

   8

   Set this to false to disable IPv4 addressing for the VF attached to the bond.

   9

   Sets a minimum transmission rate, in Mbps, for the VF. This sample value sets a rate of 100 Mbps.

   • This value must be less than or equal to the maximum transmission rate.
   • Intel NICs do not support the min-tx-rate parameter. For more information, see BZ#1772847.

   10

   Sets a maximum transmission rate, in Mbps, for the VF. This sample value sets a rate of 200 Mbps.

   11

   Sets the desired bond mode.

   12

   Sets the preferred port of the bonding interface. The bond uses the primary device as the first device of the bonding interfaces. The bond does not abandon the primary device interface unless it fails. This setting is particularly useful when one NIC in the bonding interface is faster and, therefore, able to handle a bigger load. This setting is only valid when the bonding interface is in active-backup mode (mode 1) and balance-tlb (mode 5).

   13

   Sets a static IP address for the bond interface. This is the node IP address.

   14

   Sets bond0 as the gateway for the default route.

   Important

   After deploying the cluster, you cannot change the networkConfig configuration setting of the install-config.yaml file to make changes to the host network interface. Use the Kubernetes NMState Operator to make changes to the host network interface after deployment.

1. Create the OpenShift Container Platform manifests.

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
   WARNING Discarding the OpenShift Manifest that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

Procedure

1. Install Butane on your installation host by using the following command:

   $ sudo dnf -y install butane

2. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.14.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all worker nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

3. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

4. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the worker nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.14.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

5. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

Procedure

1. Change to the directory storing the install-config.yaml file:

   $ cd ~/clusterconfigs

2. Switch to the manifests subdirectory:

   $ cd manifests

3. Create a file named cluster-network-avoid-workers-99-config.yaml:

   $ touch cluster-network-avoid-workers-99-config.yaml

4. Open the cluster-network-avoid-workers-99-config.yaml file in an editor and enter a custom resource (CR) that describes the Operator configuration:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: 50-worker-fix-ipi-rwn
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/kubernetes/manifests/keepalived.yaml
             mode: 0644
             contents:
               source: data:,

   This manifest places the ingressVIP virtual IP address on the control plane nodes. Additionally, this manifest deploys the following processes on the control plane nodes only:

   • openshift-ingress-operator
   • keepalived
5. Save the cluster-network-avoid-workers-99-config.yaml file.

6. Create a manifests/cluster-ingress-default-ingresscontroller.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/master: ""

7. Consider backing up the manifests directory. The installer deletes the manifests/ directory when creating the cluster.

8. Modify the cluster-scheduler-02-config.yml manifest to make the control plane nodes schedulable by setting the mastersSchedulable field to true. Control plane nodes are not schedulable by default. For example:

   $ sed -i "s;mastersSchedulable: false;mastersSchedulable: true;g" clusterconfigs/manifests/cluster-scheduler-02-config.yml

   Note

   If control plane nodes are not schedulable after completing this procedure, deploying the cluster will fail.

Procedure

1. Create a router-replicas.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: <num-of-router-pods>
     endpointPublishingStrategy:
       type: HostNetwork
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""

   Note

   Replace <num-of-router-pods> with an appropriate value. If working with just one worker node, set replicas: to 1. If working with more than 3 worker nodes, you can increase replicas: from the default value 2 as appropriate.

2. Save and copy the router-replicas.yaml file to the clusterconfigs/openshift directory:

   $ cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml

Procedure

1. Create the manifests.

2. Modify the BareMetalHost resource file corresponding to the node:

   $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml

3. Add the BIOS configuration to the spec section of the BareMetalHost resource:

   spec:
     firmware:
       simultaneousMultithreadingEnabled: true
       sriovEnabled: true
       virtualizationEnabled: true

   Note

   Red Hat supports three BIOS configurations. Only servers with BMC type irmc are supported. Other types of servers are currently not supported.

4. Create the cluster.

Procedure

1. Create the manifests.

2. Modify the BareMetalHost resource corresponding to the node:

   $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml

   Note

   The following example uses a hardware RAID configuration because OpenShift Container Platform 4.14 does not support software RAID.

   1. If you added a specific RAID configuration to the spec section, this causes the node to delete the original RAID configuration in the preparing phase and perform a specified configuration on the RAID. For example:

      spec:
        raid:
          hardwareRAIDVolumes:
          - level: "0" 1
            name: "sda"
            numberOfPhysicalDisks: 1
            rotational: true
            sizeGibibytes: 0

      1

      level is a required field, and the others are optional fields.

   2. If you added an empty RAID configuration to the spec section, the empty configuration causes the node to delete the original RAID configuration during the preparing phase, but does not perform a new configuration. For example:

      spec:
        raid:
          hardwareRAIDVolumes: []

   3. If you do not add a raid field in the spec section, the original RAID configuration is not deleted, and no new configuration will be performed.
3. Create the cluster.

1. Create a custom-partitions.yaml file and include a MachineConfig object that contains your partition layout:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: primary
     name: 10_primary_storage_config
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         disks:
           - device: </dev/xxyN>
             partitions:
               - label: recovery
                 startMiB: 32768
                 sizeMiB: 16384
         filesystems:
           - device: /dev/disk/by-partlabel/recovery
             label: recovery
             format: xfs

2. Save and copy the custom-partitions.yaml file to the clusterconfigs/openshift directory:

   $ cp ~/<MachineConfig_manifest> ~/clusterconfigs/openshift

Procedure

1. Open the firewall port on the registry node:

   $ sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent

   $ sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent

   $ sudo firewall-cmd --reload

2. Install the required packages for the registry node:

   $ sudo yum -y install python3 podman httpd httpd-tools jq

3. Create the directory structure where the repository information will be held:

   $ sudo mkdir -p /opt/registry/{auth,certs,data}

Procedure

1. Review the OpenShift Container Platform downloads page to determine the version of OpenShift Container Platform that you want to install and determine the corresponding tag on the Repository Tags page.

2. Set the required environment variables:

   1. Export the release version:

      $ OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   5. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

   6. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   7. Export the type of architecture for your cluster:

      $ ARCHITECTURE=<cluster_architecture> 1

      1

      Specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.

   8. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Mirror the version images to the mirror registry:

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Review the images and configuration manifests to mirror:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

     3. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

     4. Mirror the images to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

     5. Take the media to the restricted network environment and upload the images to the local container registry.

        $ oc image mirror -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

   • If the local container registry is connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

        This command pulls the release information as a digest, and its output includes the imageContentSources data that you require when you install your cluster.

     2. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

        Note

        The image name gets patched to Quay.io during the mirroring process, and the podman images will show Quay.io in the registry on the bootstrap virtual machine.

4. To create the installation program that is based on the content that you mirrored, extract it and pin it to the release:

   • If your mirror host does not have internet access, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"

   • If the local container registry is connected to the mirror host, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

     Important

     To ensure that you use the correct images for the version of OpenShift Container Platform that you selected, you must extract the installation program from the mirrored content.

     You must perform this step on a machine with an active internet connection.

     If you are in a disconnected environment, use the --image flag as part of must-gather and point to the payload image.

5. For clusters using installer-provisioned infrastructure, run the following command:

   $ openshift-baremetal-install

Procedure

1. Add the disconnected registry node’s certificate to the install-config.yaml file:

   $ echo "additionalTrustBundle: |" >> install-config.yaml

   The certificate should follow the "additionalTrustBundle: |" line and be properly indented, usually by two spaces.

   $ sed -e 's/^/  /' /opt/registry/certs/domain.crt >> install-config.yaml

2. Add the mirror information for the registry to the install-config.yaml file:

   $ echo "imageContentSources:" >> install-config.yaml

   $ echo "- mirrors:" >> install-config.yaml

   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml

   Replace registry.example.com with the registry’s fully qualified domain name.

   $ echo "  source: quay.io/openshift-release-dev/ocp-release" >> install-config.yaml

   $ echo "- mirrors:" >> install-config.yaml

   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml

   Replace registry.example.com with the registry’s fully qualified domain name.

   $ echo "  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev" >> install-config.yaml

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. To check the container logs, execute the following:

   [core@localhost ~]$ sudo podman logs -f <container_name>

   Replace <container_name> with one of ironic or ironic-inspector. If you encounter an issue where the control plane nodes are not booting up from PXE, check the ironic pod. The ironic pod contains information about the attempt to boot the cluster nodes, because it attempts to log in to the node over IPMI.

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. Check the status of the coreos-downloader container within the bootstrap VM by running the following command:

   [core@localhost ~]$ sudo podman logs -f coreos-downloader

   If the bootstrap VM cannot access the URL to the images, use the curl command to verify that the VM can access the images.

3. To inspect the bootkube logs that indicate if all the containers launched during the deployment phase, execute the following:

   [core@localhost ~]$ journalctl -xe

   [core@localhost ~]$ journalctl -b -f -u bootkube.service

4. Verify all the pods, including dnsmasq, mariadb, httpd, and ironic, are running:

   [core@localhost ~]$ sudo podman ps

5. If there are issues with the pods, check the logs of the containers with issues. To check the logs of the ironic service, run the following command:

   [core@localhost ~]$ sudo podman logs ironic

Procedure

1. Inspect the pods in the openshift-network-operator namespace:

   $ oc get all -n openshift-network-operator

   NAME                                    READY STATUS            RESTARTS   AGE
   pod/network-operator-69dfd7b577-bg89v   0/1   ContainerCreating 0          149m

2. On the provisioner node, determine that the network configuration exists:

   $ kubectl get network.config.openshift.io cluster -oyaml

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNetwork:
     - 172.30.0.0/16
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     networkType: OVNKubernetes

   If it does not exist, the installer did not create it. To determine why the installer did not create it, execute the following:

   $ openshift-install create manifests

3. Check that the network-operator is running:

   $ kubectl -n openshift-network-operator get pods

4. Retrieve the logs:

   $ kubectl -n openshift-network-operator logs -l "name=network-operator"

   On high availability clusters with three or more control plane (master) nodes, the Operator will perform leader election and all other Operators will sleep. For additional details, see Troubleshooting.

1. Ensure the reserved IPv6 addresses reside outside the DHCP range.

2. In the IP address reservation on the DHCP server, ensure the reservation specifies the correct DHCP Unique Identifier (DUID). For example:

   # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC
   id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6]

3. Ensure that route announcements are working.
4. Ensure that the DHCP server is listening on the required interfaces serving the IP address ranges.

Procedure

1. Retrieve the hostname:

   [core@master-X ~]$ hostname

   If the hostname is localhost, proceed with the following steps.

   Note

   Where X is the control plane node number.

2. Force the cluster node to renew the DHCP lease:

   [core@master-X ~]$ sudo nmcli con up "<bare_metal_nic>"

   Replace <bare_metal_nic> with the wired connection corresponding to the baremetal network.

3. Check hostname again:

   [core@master-X ~]$ hostname

4. If the hostname is still localhost.localdomain, restart NetworkManager:

   [core@master-X ~]$ sudo systemctl restart NetworkManager

5. If the hostname is still localhost.localdomain, wait a few minutes and check again. If the hostname remains localhost.localdomain, repeat the previous steps.

6. Restart the nodeip-configuration service:

   [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service

   This service will reconfigure the kubelet service with the correct hostname references.

7. Reload the unit files definition since the kubelet changed in the previous step:

   [core@master-X ~]$ sudo systemctl daemon-reload

8. Restart the kubelet service:

   [core@master-X ~]$ sudo systemctl restart kubelet.service

9. Ensure kubelet booted with the correct hostname:

   [core@master-X ~]$ sudo journalctl -fu kubelet.service

Addressing a csr

1. Get CSRs on the cluster:

   $ oc get csr

2. Verify if a pending csr contains Subject Name: localhost.localdomain:

   $ oc get csr <pending_csr> -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text

3. Remove any csr that contains Subject Name: localhost.localdomain:

   $ oc delete csr <wrong_csr>

1. Get the route:

   $ oc get route oauth-openshift

2. Check the service endpoint:

   $ oc get svc oauth-openshift

   NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   oauth-openshift   ClusterIP   172.30.19.162   <none>        443/TCP   59m

3. Attempt to reach the service from a control plane (master) node:

   [core@master0 ~]$ curl -k https://172.30.19.162

   {
     "kind": "Status",
     "apiVersion": "v1",
     "metadata": {
     },
     "status": "Failure",
     "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
     "reason": "Forbidden",
     "details": {
     },
     "code": 403

4. Identify the authentication-operator errors from the provisioner node:

   $ oc logs deployment/authentication-operator -n openshift-authentication-operator

   Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting"

Solution

1. Ensure that the cluster name for every deployment is unique, ensuring no conflict.
2. Turn off all the rogue nodes which are not part of the cluster deployment that are using the same cluster name. Otherwise, the authentication pod of the OpenShift Container Platform cluster might never start successfully.

Procedure

1. Connect to the node where the Ignition configuration failed:

   Failed Units: 1
     machine-config-daemon-firstboot.service

2. Restart the machine-config-daemon-firstboot service:

   [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service

Procedure

1. Check for differences in the AGE of the cluster nodes. For example:

   $ oc get nodes

   NAME                         STATUS   ROLES    AGE   VERSION
   master-0.cloud.example.com   Ready    master   145m   v1.27.3
   master-1.cloud.example.com   Ready    master   135m   v1.27.3
   master-2.cloud.example.com   Ready    master   145m   v1.27.3
   worker-2.cloud.example.com   Ready    worker   100m   v1.27.3

2. Check for inconsistent timing delays due to clock drift. For example:

   $ oc get bmh -n openshift-machine-api

   master-1   error registering master-1  ipmi://<out_of_band_ip>

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 18:20:02 UTC
              Universal time: Tue 2020-03-10 18:20:02 UTC
                    RTC time: Tue 2020-03-10 18:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: no
                 NTP service: active
             RTC in local TZ: no

Addressing clock drift in existing clusters

1. Create a Butane config file including the contents of the chrony.conf file to be delivered to the nodes. In the following example, create 99-master-chrony.bu to add the file to the control plane nodes. You can modify the file for worker nodes or repeat this procedure for the worker role.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.14.0
   metadata:
     name: 99-master-chrony
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           server <NTP_server> iburst 1
           stratumweight 0
           driftfile /var/lib/chrony/drift
           rtcsync
           makestep 10 3
           bindcmdaddress 127.0.0.1
           bindcmdaddress ::1
           keyfile /etc/chrony.keys
           commandkey 1
           generatecommandkey
           noclientlog
           logchange 0.5
           logdir /var/log/chrony

   1

   Replace <NTP_server> with the IP address of the NTP server.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony.yaml, containing the configuration to be delivered to the nodes:

   $ butane 99-master-chrony.bu -o 99-master-chrony.yaml

3. Apply the MachineConfig object file:

   $ oc apply -f 99-master-chrony.yaml

4. Ensure the System clock synchronized value is yes:

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 19:10:02 UTC
              Universal time: Tue 2020-03-10 19:10:02 UTC
                    RTC time: Tue 2020-03-10 19:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: yes
                 NTP service: active
             RTC in local TZ: no

   To setup clock synchronization prior to deployment, generate the manifest files and add this file to the openshift directory. For example:

   $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml

   Then, continue to create the cluster.

Procedure

1. Configure the HAProxy Ingress Controller, so that you can enable access to the cluster from your load balancer on ports 6443, 443, and 80:

   Example HAProxy configuration

   #...
   listen my-cluster-api-6443
       bind 192.168.1.100:6443
       mode tcp
       balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /readyz
     http-check expect status 200
       server my-cluster-master-2 192.168.1.101:6443 check inter 10s rise 2 fall 2
       server my-cluster-master-0 192.168.1.102:6443 check inter 10s rise 2 fall 2
       server my-cluster-master-1 192.168.1.103:6443 check inter 10s rise 2 fall 2
   
   listen my-cluster-machine-config-api-22623
       bind 192.168.1.100:22623
       mode tcp
       balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /healthz
     http-check expect status 200
       server my-cluster-master-2 192.168.1.101:22623 check inter 10s rise 2 fall 2
       server my-cluster-master-0 192.168.1.102:22623 check inter 10s rise 2 fall 2
       server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
   
   listen my-cluster-apps-443
           bind 192.168.1.100:443
           mode tcp
           balance roundrobin
       option httpchk
       http-check connect
       http-check send meth GET uri /healthz/ready
       http-check expect status 200
           server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
           server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
           server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
   
   listen my-cluster-apps-80
           bind 192.168.1.100:80
           mode tcp
           balance roundrobin
       option httpchk
       http-check connect
       http-check send meth GET uri /healthz/ready
       http-check expect status 200
           server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
           server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
           server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
   # ...

2. Use the curl CLI command to verify that the external load balancer and its resources are operational:

   1. Verify that the cluster machine configuration API is accessible to the Kubernetes API server resource, by running the following command and observing the response:

      $ curl https://<loadbalancer_ip_address>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }

   2. Verify that the cluster machine configuration API is accessible to the Machine config server resource, by running the following command and observing the output:

      $ curl -v https://<loadbalancer_ip_address>:22623/healthz --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      Content-Length: 0

   3. Verify that the controller is accessible to the Ingress Controller resource on port 80, by running the following command and observing the output:

      $ curl -I -L -H "Host: console-openshift-console.apps.<cluster_name>.<base_domain>" http://<load_balancer_front_end_IP_address>

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.ocp4.private.opequon.net/
      cache-control: no-cache

   4. Verify that the controller is accessible to the Ingress Controller resource on port 443, by running the following command and observing the output:

      $ curl -I -L --insecure --resolve console-openshift-console.apps.<cluster_name>.<base_domain>:443:<Load Balancer Front End IP Address> https://console-openshift-console.apps.<cluster_name>.<base_domain>

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

3. Configure the DNS records for your cluster to target the front-end IP addresses of the external load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer.

   Examples of modified DNS records

   <load_balancer_ip_address>  A  api.<cluster_name>.<base_domain>
   A record pointing to Load Balancer Front End

   <load_balancer_ip_address>   A apps.<cluster_name>.<base_domain>
   A record pointing to Load Balancer Front End

   Important

   DNS propagation might take some time for each DNS record to become available. Ensure that each DNS record propagates before validating each record.

4. Use the curl CLI command to verify that the external load balancer and DNS record configuration are operational:

   1. Verify that you can access the cluster API, by running the following command and observing the output:

      $ curl https://api.<cluster_name>.<base_domain>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
        }

   2. Verify that you can access the cluster machine configuration, by running the following command and observing the output:

      $ curl -v https://api.<cluster_name>.<base_domain>:22623/healthz --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      Content-Length: 0

   3. Verify that you can access each cluster application on port, by running the following command and observing the output:

      $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

   4. Verify that you can access each cluster application on port 443, by running the following command and observing the output:

      $ curl https://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private