Deploying installer-provisioned clusters on bare metal

1. Two unique virtual IP addresses.

   • One virtual IP address for the API endpoint.
   • One virtual IP address for the wildcard ingress endpoint.
2. One IP address for the provisioner node.
3. One IP address for each control plane node.
4. One IP address for each worker node, if applicable.

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:
   - name: enp2s0 1
     type: ethernet 2
     state: up 3
     ipv4:
       enabled: false 4
     ipv6:
       enabled: false
   - name: br-ex
     type: ovs-bridge
     state: up
     ipv4:
       enabled: false
       dhcp: false
     ipv6:
       enabled: false
       dhcp: false
     bridge:
       port:
       - name: enp2s0 5
       - name: br-ex
   - name: br-ex
     type: ovs-interface
     state: up
     copy-mac-from: enp2s0
     ipv4:
       enabled: true
       dhcp: true
     ipv6:
       enabled: false
       dhcp: false
   # ...

   1

   Name of the interface.

   2

   The type of ethernet.

   3

   The requested state for the interface after creation.

   4

   Disables IPv4 and IPv6 in this example.

   5

   The node NIC to which the bridge attaches.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yaml | base64 1

   1

   Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig manifest file and define a customized br-ex bridge network configuration analogous to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 1
     name: 10-br-ex-worker 2
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 3
           mode: 0644
           overwrite: true
           path: /etc/nmstate/openshift/cluster.yml
   # ...

   1

   For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. If you have a single global configuration specified in an /etc/nmstate/openshift/cluster.yml configuration file that you want to apply to all nodes in your cluster, you do not need to specify the hostname path for each node. The worker role is the default role for nodes in your cluster. The .yaml extension does not work when specifying the hostname path for each node or all nodes in the MachineConfig manifest file.

   2

   The name of the policy.

   3

   Writes the encoded base64 information to the specified path.

Procedure

1. Edit the MachineConfig CR by entering the following command:

   $ oc edit mc <machineconfig_custom_resource_name>

2. Add each compute node configuration to the CR, so that the CR can manage roles for each defined compute node in your cluster.
3. Create a Secret object named extraworker-secret that has a minimal static IP configuration.

4. Apply the extraworker-secret secret to each node in your cluster by entering the following command. This step provides each compute node access to the Ignition config file.

   $ oc apply -f ./extraworker-secret.yaml

5. Create a BareMetalHost resource and specify the network secret in the preprovisioningNetworkDataName parameter:

   Example BareMetalHost resource with an attached network secret

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   spec:
   # ...
     preprovisioningNetworkDataName: ostest-extraworker-0-network-config-secret
   # ...

6. To manage the BareMetalHost object within the openshift-machine-api namespace of your cluster, change to the namespace by entering the following command:

   $ oc project openshift-machine-api

7. Get the machine sets:

   $ oc get machinesets

8. Scale each machine set by entering the following command. You must run this command for each machine set.

   $ oc scale machineset <machineset_name> --replicas=<n> 1

   1

   Where <machineset_name> is the name of the machine set and <n> is the number of compute nodes.

Procedure

1. Configure the HAProxy Ingress Controller, so that you can enable access to the cluster from your load balancer on ports 6443, 22623, 443, and 80. Depending on your needs, you can specify the IP address of a single subnet or IP addresses from multiple subnets in your HAProxy configuration.

   Example HAProxy configuration with one listed subnet

   # ...
   listen my-cluster-api-6443
       bind 192.168.1.100:6443
       mode tcp
       balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /readyz
     http-check expect status 200
       server my-cluster-master-2 192.168.1.101:6443 check inter 10s rise 2 fall 2
       server my-cluster-master-0 192.168.1.102:6443 check inter 10s rise 2 fall 2
       server my-cluster-master-1 192.168.1.103:6443 check inter 10s rise 2 fall 2
   
   listen my-cluster-machine-config-api-22623
       bind 192.168.1.100:22623
       mode tcp
       balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /healthz
     http-check expect status 200
       server my-cluster-master-2 192.168.1.101:22623 check inter 10s rise 2 fall 2
       server my-cluster-master-0 192.168.1.102:22623 check inter 10s rise 2 fall 2
       server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
   
   listen my-cluster-apps-443
       bind 192.168.1.100:443
       mode tcp
       balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /healthz/ready
     http-check expect status 200
       server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
       server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
       server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
   
   listen my-cluster-apps-80
      bind 192.168.1.100:80
      mode tcp
      balance roundrobin
     option httpchk
     http-check connect
     http-check send meth GET uri /healthz/ready
     http-check expect status 200
       server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
       server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
       server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
   # ...

   Example HAProxy configuration with multiple listed subnets

   # ...
   listen api-server-6443
       bind *:6443
       mode tcp
         server master-00 192.168.83.89:6443 check inter 1s
         server master-01 192.168.84.90:6443 check inter 1s
         server master-02 192.168.85.99:6443 check inter 1s
         server bootstrap 192.168.80.89:6443 check inter 1s
   
   listen machine-config-server-22623
       bind *:22623
       mode tcp
         server master-00 192.168.83.89:22623 check inter 1s
         server master-01 192.168.84.90:22623 check inter 1s
         server master-02 192.168.85.99:22623 check inter 1s
         server bootstrap 192.168.80.89:22623 check inter 1s
   
   listen ingress-router-80
       bind *:80
       mode tcp
       balance source
         server worker-00 192.168.83.100:80 check inter 1s
         server worker-01 192.168.83.101:80 check inter 1s
   
   listen ingress-router-443
       bind *:443
       mode tcp
       balance source
         server worker-00 192.168.83.100:443 check inter 1s
         server worker-01 192.168.83.101:443 check inter 1s
   
   listen ironic-api-6385
       bind *:6385
       mode tcp
       balance source
         server master-00 192.168.83.89:6385 check inter 1s
         server master-01 192.168.84.90:6385 check inter 1s
         server master-02 192.168.85.99:6385 check inter 1s
         server bootstrap 192.168.80.89:6385 check inter 1s
   
   listen inspector-api-5050
       bind *:5050
       mode tcp
       balance source
         server master-00 192.168.83.89:5050 check inter 1s
         server master-01 192.168.84.90:5050 check inter 1s
         server master-02 192.168.85.99:5050 check inter 1s
         server bootstrap 192.168.80.89:5050 check inter 1s
   # ...

2. Use the curl CLI command to verify that the user-managed load balancer and its resources are operational:

   1. Verify that the cluster machine configuration API is accessible to the Kubernetes API server resource, by running the following command and observing the response:

      $ curl https://<loadbalancer_ip_address>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }

   2. Verify that the cluster machine configuration API is accessible to the Machine config server resource, by running the following command and observing the output:

      $ curl -v https://<loadbalancer_ip_address>:22623/healthz --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      Content-Length: 0

   3. Verify that the controller is accessible to the Ingress Controller resource on port 80, by running the following command and observing the output:

      $ curl -I -L -H "Host: console-openshift-console.apps.<cluster_name>.<base_domain>" http://<load_balancer_front_end_IP_address>

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.ocp4.private.opequon.net/
      cache-control: no-cache

   4. Verify that the controller is accessible to the Ingress Controller resource on port 443, by running the following command and observing the output:

      $ curl -I -L --insecure --resolve console-openshift-console.apps.<cluster_name>.<base_domain>:443:<Load Balancer Front End IP Address> https://console-openshift-console.apps.<cluster_name>.<base_domain>

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

3. Configure the DNS records for your cluster to target the front-end IP addresses of the user-managed load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer.

   Examples of modified DNS records

   <load_balancer_ip_address>  A  api.<cluster_name>.<base_domain>
   A record pointing to Load Balancer Front End

   <load_balancer_ip_address>   A apps.<cluster_name>.<base_domain>
   A record pointing to Load Balancer Front End

   Important

   DNS propagation might take some time for each DNS record to become available. Ensure that each DNS record propagates before validating each record.

4. For your OpenShift Container Platform cluster to use the user-managed load balancer, you must specify the following configuration in your cluster’s install-config.yaml file:

   # ...
   platform:
     baremetal:
       loadBalancer:
         type: UserManaged 1
         apiVIPs:
         - <api_ip> 2
         ingressVIPs:
         - <ingress_ip> 3
   # ...

   1

   Set UserManaged for the type parameter to specify a user-managed load balancer for your cluster. The parameter defaults to OpenShiftManagedDefault, which denotes the default internal load balancer. For services defined in an openshift-kni-infra namespace, a user-managed load balancer can deploy the coredns service to pods in your cluster but ignores keepalived and haproxy services.

   2

   Required parameter when you specify a user-managed load balancer. Specify the user-managed load balancer’s public IP address, so that the Kubernetes API can communicate with the user-managed load balancer.

   3

   Required parameter when you specify a user-managed load balancer. Specify the user-managed load balancer’s public IP address, so that the user-managed load balancer can manage ingress traffic for your cluster.

Verification

1. Use the curl CLI command to verify that the user-managed load balancer and DNS record configuration are operational:

   1. Verify that you can access the cluster API, by running the following command and observing the output:

      $ curl https://api.<cluster_name>.<base_domain>:6443/version --insecure

      If the configuration is correct, you receive a JSON object in response:

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
        }

   2. Verify that you can access the cluster machine configuration, by running the following command and observing the output:

      $ curl -v https://api.<cluster_name>.<base_domain>:22623/healthz --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      Content-Length: 0

   3. Verify that you can access each cluster application on port, by running the following command and observing the output:

      $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

   4. Verify that you can access each cluster application on port 443, by running the following command and observing the output:

      $ curl https://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure

      If the configuration is correct, the output from the command shows the following response:

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private

1. Configure install-config.yaml. Change the appropriate variables to match the environment, including pullSecret and sshKey:

   apiVersion: v1
   baseDomain: <domain>
   metadata:
     name: <cluster_name>
   networking:
     machineNetwork:
     - cidr: <public_cidr>
     networkType: OVNKubernetes
   compute:
   - name: worker
     replicas: 2 1
   controlPlane:
     name: master
     replicas: 3
     platform:
       baremetal: {}
   platform:
     baremetal:
       apiVIPs:
         - <api_ip>
       ingressVIPs:
         - <wildcard_ip>
       provisioningNetworkCIDR: <CIDR>
       bootstrapExternalStaticIP: <bootstrap_static_ip_address> 2
       bootstrapExternalStaticGateway: <bootstrap_static_gateway> 3
       bootstrapExternalStaticDNS: <bootstrap_static_dns> 4
       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: ipmi://<out_of_band_ip> 5
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>" 6
         - name: <openshift_master_1>
           role: master
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
         - name: <openshift_master_2>
           role: master
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
         - name: <openshift_worker_0>
           role: worker
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
         - name: <openshift_worker_1>
           role: worker
           bmc:
             address: ipmi://<out_of_band_ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1_mac_address>
           rootDeviceHints:
            deviceName: "<installation_disk_drive_path>"
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_pub_key>'

   1

   Scale the compute machines based on the number of compute nodes that are part of the OpenShift Container Platform cluster. Valid options for the replicas value are 0 and integers greater than or equal to 2. Set the number of replicas to 0 to deploy a three-node cluster, which contains only three control plane machines. A three-node cluster is a smaller, more resource-efficient cluster that can be used for testing, development, and production. You cannot install the cluster with only one compute node.

   2

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticIP configuration setting to specify the static IP address of the bootstrap VM when there is no DHCP server on the bare-metal network.

   3

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticGateway configuration setting to specify the gateway IP address for the bootstrap VM when there is no DHCP server on the bare-metal network.

   4

   When deploying a cluster with static IP addresses, you must set the bootstrapExternalStaticDNS configuration setting to specify the DNS address for the bootstrap VM when there is no DHCP server on the bare-metal network.

   5

   See the BMC addressing sections for more options.

   6

   To set the path to the installation disk drive, enter the kernel name of the disk. For example, /dev/sda.

   Important

   Because the disk discovery order is not guaranteed, the kernel name of the disk can change across booting options for machines with multiple disks. For example, /dev/sda becomes /dev/sdb and vice versa. To avoid this issue, you must use persistent disk attributes, such as the disk World Wide Name (WWN) or /dev/disk/by-path/. It is recommended to use the /dev/disk/by-path/<device_path> link to the storage location. To use the disk WWN, replace the deviceName parameter with the wwnWithExtension parameter. Depending on the parameter that you use, enter either of the following values:

   • The disk name. For example, /dev/sda, or /dev/disk/by-path/.
   • The disk WWN. For example, "0x64cd98f04fde100024684cf3034da5c2". Ensure that you enter the disk WWN value within quotes so that it is used as a string value and not a hexadecimal value.

   Failure to meet these requirements for the rootDeviceHints parameter might result in the following error:

   ironic-inspector inspection failed: No disks satisfied root device hints

   Note

   Before OpenShift Container Platform 4.12, the cluster installation program only accepted an IPv4 address or an IPv6 address for the apiVIP and ingressVIP configuration settings. In OpenShift Container Platform 4.12 and later, these configuration settings are deprecated. Instead, use a list format in the apiVIPs and ingressVIPs configuration settings to specify IPv4 addresses, IPv6 addresses, or both IP address formats.

2. Create a directory to store the cluster configuration:

   $ mkdir ~/clusterconfigs

3. Copy the install-config.yaml file to the new directory:

   $ cp install-config.yaml ~/clusterconfigs

4. Ensure all bare metal nodes are powered off prior to installing the OpenShift Container Platform cluster:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

5. Remove old bootstrap resources if any are left over from a previous deployment attempt:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

Procedure

1. Set the IP address or hostname of the BMC by running the following command:

   $ export SERVER=<ip_address> 1

   1

   Replace <ip_address> with the IP address or hostname of the BMC.

2. Set the ID of the system by running the following command:

   $ export SystemID=<system_id> 1

   1

   Replace <system_id> with the system ID. For example, System.Embedded.1 or 1. See the following vendor-specific BMC sections for details.

List of Redfish APIs

1. Check power on support by running the following command:

   $ curl -u $USER:$PASS -X POST -H'Content-Type: application/json' -H'Accept: application/json' -d '{"ResetType": "On"}' https://$SERVER/redfish/v1/Systems/$SystemID/Actions/ComputerSystem.Reset

2. Check power off support by running the following command:

   $ curl -u $USER:$PASS -X POST -H'Content-Type: application/json' -H'Accept: application/json' -d '{"ResetType": "ForceOff"}' https://$SERVER/redfish/v1/Systems/$SystemID/Actions/ComputerSystem.Reset

3. Check the temporary boot implementation that uses pxe by running the following command:

   $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json"  https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideTarget": "pxe", "BootSourceOverrideEnabled": "Once"}}

4. Check the status of setting the BIOS boot mode that uses Legacy or UEFI by running the following command:

   $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json"  https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideMode":"UEFI"}}

List of Redfish virtual media APIs

1. Check the ability to set the temporary boot device that uses cd or dvd by running the following command:

   $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideTarget": "cd", "BootSourceOverrideEnabled": "Once"}}'

2. Check the ability to mount virtual media by running the following command:

   $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" -H "If-Match: *" https://$Server/redfish/v1/Managers/$ManagerID/VirtualMedia/$VmediaId -d '{"Image": "https://example.com/test.iso", "TransferProtocolType": "HTTPS", "UserName": "", "Password":""}'

Procedure

1. Add proxy values under the proxy key mapping:

   apiVersion: v1
   baseDomain: <domain>
   proxy:
     httpProxy: http://USERNAME:PASSWORD@proxy.example.com:PORT
     httpsProxy: https://USERNAME:PASSWORD@proxy.example.com:PORT
     noProxy: <WILDCARD_OF_DOMAIN>,<PROVISIONING_NETWORK/CIDR>,<BMC_ADDRESS_RANGE/CIDR>

   The following is an example of noProxy with values.

   noProxy: .example.com,172.22.0.0/24,10.10.0.0/24

2. With a proxy enabled, set the appropriate values of the proxy in the corresponding key/value pair.

   Key considerations:

   • If the proxy does not have an HTTPS proxy, change the value of httpsProxy from https:// to http://.
   • If the cluster uses a provisioning network, include it in the noProxy setting, otherwise the installation program fails.
   • Set all of the proxy settings as environment variables within the provisioner node. For example, HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

Procedure

1. Optional: Consider testing the NMState syntax with nmstatectl gc before including it in the install-config.yaml file, because the installer will not check the NMState YAML syntax.

   Note

   Errors in the YAML syntax might result in a failure to apply the network configuration. Additionally, maintaining the validated YAML syntax is useful when applying changes using Kubernetes NMState after deployment or when expanding the cluster.

   1. Create an NMState YAML file:

      interfaces:
      - name: <nic1_name> 1
        type: ethernet
        state: up
        ipv4:
          address:
          - ip: <ip_address> 2
            prefix-length: 24
          enabled: true
      dns-resolver:
        config:
          server:
          - <dns_ip_address> 3
      routes:
        config:
        - destination: 0.0.0.0/0
          next-hop-address: <next_hop_ip_address> 4
          next-hop-interface: <next_hop_nic1_name> 5

      1 2 3 4 5

      Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

   2. Test the configuration file by running the following command:

      $ nmstatectl gc <nmstate_yaml_file>

      Replace <nmstate_yaml_file> with the configuration file name.

2. Use the networkConfig configuration setting by adding the NMState configuration to hosts within the install-config.yaml file:

       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: null
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           rootDeviceHints:
             deviceName: "/dev/sda"
           networkConfig: 1
             interfaces:
             - name: <nic1_name> 2
               type: ethernet
               state: up
               ipv4:
                 address:
                 - ip: <ip_address> 3
                   prefix-length: 24
                 enabled: true
             dns-resolver:
               config:
                 server:
                 - <dns_ip_address> 4
             routes:
               config:
               - destination: 0.0.0.0/0
                 next-hop-address: <next_hop_ip_address> 5
                 next-hop-interface: <next_hop_nic1_name> 6

   1

   Add the NMState YAML syntax to configure the host interfaces.

   2 3 4 5 6

   Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

   Important

   After deploying the cluster, you cannot modify the networkConfig configuration setting of install-config.yaml file to make changes to the host network interface. Use the Kubernetes NMState Operator to make changes to the host network interface after deployment.

Procedure

1. Add the subnets to the machineNetwork in the install-config.yaml file when using static IP addresses:

   networking:
     machineNetwork:
     - cidr: 10.0.0.0/24
     - cidr: 192.168.0.0/24
     networkType: OVNKubernetes

2. Add the gateway and DNS configuration to the networkConfig parameter of each edge compute node using NMState syntax when using a static IP address or advanced networking such as bonds:

   networkConfig:
     interfaces:
     - name: <interface_name> 1
       type: ethernet
       state: up
       ipv4:
         enabled: true
         dhcp: false
         address:
         - ip: <node_ip> 2
           prefix-length: 24
         gateway: <gateway_ip> 3
     dns-resolver:
       config:
         server:
         - <dns_ip> 4

   1

   Replace <interface_name> with the interface name.

   2

   Replace <node_ip> with the IP address of the node.

   3

   Replace <gateway_ip> with the IP address of the gateway.

   4

   Replace <dns_ip> with the IP address of the DNS server.

Procedure

1. Optional: Consider testing the NMState YAML syntax with the nmstatectl gc command before including it in the install-config.yaml file because the installation program will not check the NMState YAML syntax.

   1. Create an NMState YAML file:

      interfaces:
      - name: eth0
        ipv6:
          addr-gen-mode: <address_mode> 1

      1

      Replace <address_mode> with the type of address generation mode required for IPv6 addresses in the cluster. Valid values are eui64, stable-privacy, or random.

   2. Test the configuration file by running the following command:

      $ nmstatectl gc <nmstate_yaml_file> 1

      1

      Replace <nmstate_yaml_file> with the name of the test configuration file.

2. Add the NMState configuration to the hosts.networkConfig section within the install-config.yaml file:

       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: null
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           rootDeviceHints:
             deviceName: "/dev/sda"
           networkConfig:
             interfaces:
             - name: eth0
               ipv6:
                 addr-gen-mode: <address_mode> 1
   ...

   1

   Replace <address_mode> with the type of address generation mode required for IPv6 addresses in the cluster. Valid values are eui64, stable-privacy, or random.

Procedure

1. Add the NMState configuration to the networkConfig field to hosts within the install-config.yaml file:

       hosts:
         - name: worker-0
           role: worker
           bmc:
             address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
             username: <user>
             password: <password>
             disableCertificateVerification: false
           bootMACAddress: <NIC1_mac_address>
           bootMode: UEFI
           networkConfig: 1
             interfaces: 2
              - name: eno1 3
                type: ethernet 4
                state: up
                mac-address: 0c:42:a1:55:f3:06
                ipv4:
                  enabled: true
                  dhcp: false 5
                ethernet:
                  sr-iov:
                    total-vfs: 2 6
                ipv6:
                  enabled: false
                  dhcp: false
              - name: sriov:eno1:0
                type: ethernet
                state: up 7
                ipv4:
                  enabled: false 8
                ipv6:
                  enabled: false
              - name: sriov:eno1:1
                type: ethernet
                state: down
              - name: eno2
                type: ethernet
                state: up
                mac-address: 0c:42:a1:55:f3:07
                ipv4:
                  enabled: true
                ethernet:
                  sr-iov:
                    total-vfs: 2
                ipv6:
                  enabled: false
              - name: sriov:eno2:0
                type: ethernet
                state: up
                ipv4:
                  enabled: false
                ipv6:
                  enabled: false
              - name: sriov:eno2:1
                type: ethernet
                state: down
              - name: bond0
                type: bond
                state: up
                min-tx-rate: 100 9
                max-tx-rate: 200 10
                link-aggregation:
                  mode: active-backup 11
                  options:
                    primary: sriov:eno1:0 12
                  port:
                    - sriov:eno1:0
                    - sriov:eno2:0
                ipv4:
                  address:
                    - ip: 10.19.16.57 13
                      prefix-length: 23
                  dhcp: false
                  enabled: true
                ipv6:
                  enabled: false
             dns-resolver:
               config:
                 server:
                   - 10.11.5.160
                   - 10.2.70.215
             routes:
               config:
                 - destination: 0.0.0.0/0
                   next-hop-address: 10.19.17.254
                   next-hop-interface: bond0 14
                   table-id: 254

   1

   The networkConfig field has information about the network configuration of the host, with subfields including interfaces, dns-resolver, and routes.

   2

   The interfaces field is an array of network interfaces defined for the host.

   3

   The name of the interface.

   4

   The type of interface. This example creates a ethernet interface.

   5

   Set this to `false to disable DHCP for the physical function (PF) if it is not strictly required.

   6

   Set to the number of SR-IOV virtual functions (VFs) to instantiate.

   7

   Set this to up.

   8

   Set this to false to disable IPv4 addressing for the VF attached to the bond.

   9

   Sets a minimum transmission rate, in Mbps, for the VF. This sample value sets a rate of 100 Mbps.

   • This value must be less than or equal to the maximum transmission rate.
   • Intel NICs do not support the min-tx-rate parameter. For more information, see BZ#1772847.

   10

   Sets a maximum transmission rate, in Mbps, for the VF. This sample value sets a rate of 200 Mbps.

   11

   Sets the desired bond mode.

   12

   Sets the preferred port of the bonding interface. The bond uses the primary device as the first device of the bonding interfaces. The bond does not abandon the primary device interface unless it fails. This setting is particularly useful when one NIC in the bonding interface is faster and, therefore, able to handle a bigger load. This setting is only valid when the bonding interface is in active-backup mode (mode 1) and balance-tlb (mode 5).

   13

   Sets a static IP address for the bond interface. This is the node IP address.

   14

   Sets bond0 as the gateway for the default route.

   Important

   After deploying the cluster, you cannot change the networkConfig configuration setting of the install-config.yaml file to make changes to the host network interface. Use the Kubernetes NMState Operator to make changes to the host network interface after deployment.

1. Create the OpenShift Container Platform manifests.

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
   WARNING Discarding the OpenShift Manifest that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

Procedure

1. Install Butane on your installation host by using the following command:

   $ sudo dnf -y install butane

2. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.17.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all compute nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

3. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

4. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the compute nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.17.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

5. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

Procedure

1. Change to the directory storing the install-config.yaml file:

   $ cd ~/clusterconfigs

2. Switch to the manifests subdirectory:

   $ cd manifests

3. Create a file named cluster-network-avoid-workers-99-config.yaml:

   $ touch cluster-network-avoid-workers-99-config.yaml

4. Open the cluster-network-avoid-workers-99-config.yaml file in an editor and enter a custom resource (CR) that describes the Operator configuration:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: 50-worker-fix-ipi-rwn
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/kubernetes/manifests/keepalived.yaml
             mode: 0644
             contents:
               source: data:,

   This manifest places the ingressVIP virtual IP address on the control plane nodes. Additionally, this manifest deploys the following processes on the control plane nodes only:

   • openshift-ingress-operator
   • keepalived
5. Save the cluster-network-avoid-workers-99-config.yaml file.

6. Create a manifests/cluster-ingress-default-ingresscontroller.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/master: ""

7. Consider backing up the manifests directory. The installer deletes the manifests/ directory when creating the cluster.

8. Modify the cluster-scheduler-02-config.yml manifest to make the control plane nodes schedulable by setting the mastersSchedulable field to true. Control plane nodes are not schedulable by default. For example:

   $ sed -i "s;mastersSchedulable: false;mastersSchedulable: true;g" clusterconfigs/manifests/cluster-scheduler-02-config.yml

   Note

   If control plane nodes are not schedulable after completing this procedure, deploying the cluster will fail.

Procedure

1. Create a router-replicas.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: <num-of-router-pods>
     endpointPublishingStrategy:
       type: HostNetwork
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""

   Note

   Replace <num-of-router-pods> with an appropriate value. If working with just one compute node, set replicas: to 1. If working with more than 3 compute nodes, you can increase replicas: from the default value 2 as appropriate.

2. Save and copy the router-replicas.yaml file to the clusterconfigs/openshift directory:

   $ cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml

Procedure

1. Create the manifests.

2. Modify the BareMetalHost resource file corresponding to the node:

   $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml

3. Add the BIOS configuration to the spec section of the BareMetalHost resource:

   spec:
     firmware:
       simultaneousMultithreadingEnabled: true
       sriovEnabled: true
       virtualizationEnabled: true

   Note

   Red Hat supports three BIOS configurations. Only servers with BMC type irmc are supported. Other types of servers are currently not supported.

4. Create the cluster.

Procedure

1. Create the manifests.

2. Modify the BareMetalHost resource corresponding to the node:

   $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml

   Note

   The following example uses a hardware RAID configuration because OpenShift Container Platform 4.17 does not support software RAID.

   1. If you added a specific RAID configuration to the spec section, this causes the node to delete the original RAID configuration in the preparing phase and perform a specified configuration on the RAID. For example:

      spec:
        raid:
          hardwareRAIDVolumes:
          - level: "0" 1
            name: "sda"
            numberOfPhysicalDisks: 1
            rotational: true
            sizeGibibytes: 0

      1

      level is a required field, and the others are optional fields.

   2. If you added an empty RAID configuration to the spec section, the empty configuration causes the node to delete the original RAID configuration during the preparing phase, but does not perform a new configuration. For example:

      spec:
        raid:
          hardwareRAIDVolumes: []

   3. If you do not add a raid field in the spec section, the original RAID configuration is not deleted, and no new configuration will be performed.
3. Create the cluster.

1. Create a custom-partitions.yaml file and include a MachineConfig object that contains your partition layout:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: primary
     name: 10_primary_storage_config
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         disks:
           - device: </dev/xxyN>
             partitions:
               - label: recovery
                 startMiB: 32768
                 sizeMiB: 16384
         filesystems:
           - device: /dev/disk/by-partlabel/recovery
             label: recovery
             format: xfs

2. Save and copy the custom-partitions.yaml file to the clusterconfigs/openshift directory:

   $ cp ~/<MachineConfig_manifest> ~/clusterconfigs/openshift

Procedure

1. Open the firewall port on the registry node:

   $ sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent

   $ sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent

   $ sudo firewall-cmd --reload

2. Install the required packages for the registry node:

   $ sudo yum -y install python3 podman httpd httpd-tools jq

3. Create the directory structure where the repository information will be held:

   $ sudo mkdir -p /opt/registry/{auth,certs,data}

Procedure

1. Review the OpenShift Container Platform downloads page to determine the version of OpenShift Container Platform that you want to install and determine the corresponding tag on the Repository Tags page.

2. Set the required environment variables:

   1. Export the release version:

      $ OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   5. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

   6. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   7. Export the type of architecture for your cluster:

      $ ARCHITECTURE=<cluster_architecture> 1

      1

      Specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.

   8. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Mirror the version images to the mirror registry:

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Review the images and configuration manifests to mirror:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

     3. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

     4. Mirror the images to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

     5. Take the media to the restricted network environment and upload the images to the local container registry.

        $ oc image mirror -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

   • If the local container registry is connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

        This command pulls the release information as a digest, and its output includes the imageContentSources data that you require when you install your cluster.

     2. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

        Note

        The image name gets patched to Quay.io during the mirroring process, and the podman images will show Quay.io in the registry on the bootstrap virtual machine.

4. To create the installation program that is based on the content that you mirrored, extract it and pin it to the release:

   • If your mirror host does not have internet access, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"

   • If the local container registry is connected to the mirror host, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

     Important

     To ensure that you use the correct images for the version of OpenShift Container Platform that you selected, you must extract the installation program from the mirrored content.

     You must perform this step on a machine with an active internet connection.

     If you are in a disconnected environment, use the --image flag as part of must-gather and point to the payload image.

5. For clusters using installer-provisioned infrastructure, run the following command:

   $ openshift-baremetal-install

Procedure

1. Add the disconnected registry node’s certificate to the install-config.yaml file:

   $ echo "additionalTrustBundle: |" >> install-config.yaml

   The certificate should follow the "additionalTrustBundle: |" line and be properly indented, usually by two spaces.

   $ sed -e 's/^/  /' /opt/registry/certs/domain.crt >> install-config.yaml

2. Add the mirror information for the registry to the install-config.yaml file:

   $ echo "imageContentSources:" >> install-config.yaml

   $ echo "- mirrors:" >> install-config.yaml

   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml

   Replace registry.example.com with the registry’s fully qualified domain name.

   $ echo "  source: quay.io/openshift-release-dev/ocp-release" >> install-config.yaml

   $ echo "- mirrors:" >> install-config.yaml

   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml

   Replace registry.example.com with the registry’s fully qualified domain name.

   $ echo "  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev" >> install-config.yaml

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. To check the container logs, execute the following:

   [core@localhost ~]$ sudo podman logs -f <container_name>

   Replace <container_name> with one of ironic or ironic-inspector. If you encounter an issue where the control plane nodes are not booting up from PXE, check the ironic pod. The ironic pod contains information about the attempt to boot the cluster nodes, because it attempts to log in to the node over IPMI.

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. Check the status of the coreos-downloader container within the bootstrap VM by running the following command:

   [core@localhost ~]$ sudo podman logs -f coreos-downloader

   If the bootstrap VM cannot access the URL to the images, use the curl command to verify that the VM can access the images.

3. To inspect the bootkube logs that indicate if all the containers launched during the deployment phase, execute the following:

   [core@localhost ~]$ journalctl -xe

   [core@localhost ~]$ journalctl -b -f -u bootkube.service

4. Verify all the pods, including dnsmasq, mariadb, httpd, and ironic, are running:

   [core@localhost ~]$ sudo podman ps

5. If there are issues with the pods, check the logs of the containers with issues. To check the logs of the ironic service, run the following command:

   [core@localhost ~]$ sudo podman logs ironic

Procedure

1. Inspect the pods in the openshift-network-operator namespace:

   $ oc get all -n openshift-network-operator

   NAME                                    READY STATUS            RESTARTS   AGE
   pod/network-operator-69dfd7b577-bg89v   0/1   ContainerCreating 0          149m

2. On the provisioner node, determine that the network configuration exists:

   $ kubectl get network.config.openshift.io cluster -oyaml

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNetwork:
     - 172.30.0.0/16
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     networkType: OVNKubernetes

   If it does not exist, the installation program did not create it. To determine why the installation program did not create it, execute the following:

   $ openshift-install create manifests

3. Check that the network-operator is running:

   $ kubectl -n openshift-network-operator get pods

4. Retrieve the logs:

   $ kubectl -n openshift-network-operator logs -l "name=network-operator"

   On high availability clusters with three or more control plane nodes, the Operator will perform leader election and all other Operators will sleep. For additional details, see Troubleshooting.

1. Ensure the reserved IPv6 addresses reside outside the DHCP range.

2. In the IP address reservation on the DHCP server, ensure the reservation specifies the correct DHCP Unique Identifier (DUID). For example:

   # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC
   id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6]

3. Ensure that route announcements are working.
4. Ensure that the DHCP server is listening on the required interfaces serving the IP address ranges.

Procedure

1. Retrieve the hostname:

   [core@master-X ~]$ hostname

   If the hostname is localhost, proceed with the following steps.

   Note

   Where X is the control plane node number.

2. Force the cluster node to renew the DHCP lease:

   [core@master-X ~]$ sudo nmcli con up "<bare_metal_nic>"

   Replace <bare_metal_nic> with the wired connection corresponding to the baremetal network.

3. Check hostname again:

   [core@master-X ~]$ hostname

4. If the hostname is still localhost.localdomain, restart NetworkManager:

   [core@master-X ~]$ sudo systemctl restart NetworkManager

5. If the hostname is still localhost.localdomain, wait a few minutes and check again. If the hostname remains localhost.localdomain, repeat the previous steps.

6. Restart the nodeip-configuration service:

   [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service

   This service will reconfigure the kubelet service with the correct hostname references.

7. Reload the unit files definition since the kubelet changed in the previous step:

   [core@master-X ~]$ sudo systemctl daemon-reload

8. Restart the kubelet service:

   [core@master-X ~]$ sudo systemctl restart kubelet.service

9. Ensure kubelet booted with the correct hostname:

   [core@master-X ~]$ sudo journalctl -fu kubelet.service

Addressing a csr

1. Get CSRs on the cluster:

   $ oc get csr

2. Verify if a pending csr contains Subject Name: localhost.localdomain:

   $ oc get csr <pending_csr> -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text

3. Remove any csr that contains Subject Name: localhost.localdomain:

   $ oc delete csr <wrong_csr>

1. Get the route:

   $ oc get route oauth-openshift

2. Check the service endpoint:

   $ oc get svc oauth-openshift

   NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   oauth-openshift   ClusterIP   172.30.19.162   <none>        443/TCP   59m

3. Attempt to reach the service from a control plane (master) node:

   [core@master0 ~]$ curl -k https://172.30.19.162

   {
     "kind": "Status",
     "apiVersion": "v1",
     "metadata": {
     },
     "status": "Failure",
     "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
     "reason": "Forbidden",
     "details": {
     },
     "code": 403

4. Identify the authentication-operator errors from the provisioner node:

   $ oc logs deployment/authentication-operator -n openshift-authentication-operator

   Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting"

Solution

1. Ensure that the cluster name for every deployment is unique, ensuring no conflict.
2. Turn off all the rogue nodes which are not part of the cluster deployment that are using the same cluster name. Otherwise, the authentication pod of the OpenShift Container Platform cluster might never start successfully.

Procedure

1. Connect to the node where the Ignition configuration failed:

   Failed Units: 1
     machine-config-daemon-firstboot.service

2. Restart the machine-config-daemon-firstboot service:

   [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service

Procedure

1. Check for differences in the AGE of the cluster nodes. For example:

   $ oc get nodes

   NAME                         STATUS   ROLES    AGE   VERSION
   master-0.cloud.example.com   Ready    master   145m   v1.30.3
   master-1.cloud.example.com   Ready    master   135m   v1.30.3
   master-2.cloud.example.com   Ready    master   145m   v1.30.3
   worker-2.cloud.example.com   Ready    worker   100m   v1.30.3

2. Check for inconsistent timing delays due to clock drift. For example:

   $ oc get bmh -n openshift-machine-api

   master-1   error registering master-1  ipmi://<out_of_band_ip>

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 18:20:02 UTC
              Universal time: Tue 2020-03-10 18:20:02 UTC
                    RTC time: Tue 2020-03-10 18:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: no
                 NTP service: active
             RTC in local TZ: no

Addressing clock drift in existing clusters

1. Create a Butane config file including the contents of the chrony.conf file to be delivered to the nodes. In the following example, create 99-master-chrony.bu to add the file to the control plane nodes. You can modify the file for worker nodes or repeat this procedure for the worker role.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.17.0
   metadata:
     name: 99-master-chrony
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           server <NTP_server> iburst 1
           stratumweight 0
           driftfile /var/lib/chrony/drift
           rtcsync
           makestep 10 3
           bindcmdaddress 127.0.0.1
           bindcmdaddress ::1
           keyfile /etc/chrony.keys
           commandkey 1
           generatecommandkey
           noclientlog
           logchange 0.5
           logdir /var/log/chrony

   1

   Replace <NTP_server> with the IP address of the NTP server.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony.yaml, containing the configuration to be delivered to the nodes:

   $ butane 99-master-chrony.bu -o 99-master-chrony.yaml

3. Apply the MachineConfig object file:

   $ oc apply -f 99-master-chrony.yaml

4. Ensure the System clock synchronized value is yes:

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 19:10:02 UTC
              Universal time: Tue 2020-03-10 19:10:02 UTC
                    RTC time: Tue 2020-03-10 19:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: yes
                 NTP service: active
             RTC in local TZ: no

   To setup clock synchronization prior to deployment, generate the manifest files and add this file to the openshift directory. For example:

   $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml

   Then, continue to create the cluster.

Procedure

1. Create a NodeNetworkConfigurationPolicy (NNCP) CR and define a customized br-ex bridge network configuration. Depending on your needs, ensure that you set a masquerade IP for either the ipv4.address.ip, ipv6.address.ip, or both parameters. A masquerade IP address must match an in-use IP address block.

   Example of an NNCP CR that sets IPv6 and IPv4 masquerade IP addresses

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: worker-0-br-ex 1
   spec:
     nodeSelector:
       kubernetes.io/hostname: worker-0
       desiredState:
       interfaces:
       - name: enp2s0 2
         type: ethernet 3
         state: up 4
         ipv4:
           enabled: false 5
         ipv6:
           enabled: false
       - name: br-ex
         type: ovs-bridge
         state: up
         ipv4:
           enabled: false
           dhcp: false
         ipv6:
           enabled: false
           dhcp: false
         bridge:
           port:
           - name: enp2s0 6
           - name: br-ex
       - name: br-ex
         type: ovs-interface
         state: up
         copy-mac-from: enp2s0
         ipv4:
           enabled: true
           dhcp: true
           address:
           - ip: "169.254.169.2"
             prefix-length: 29
         ipv6:
           enabled: false
           dhcp: false
           address:
           - ip: "fd69::2"
           prefix-length: 125

   1

   Name of the policy.

   2

   Name of the interface.

   3

   The type of ethernet.

   4

   The requested state for the interface after creation.

   5

   Disables IPv4 and IPv6 in this example.

   6

   The node NIC to which the bridge is attached.

1. The BareMetalHost spec
2. The BareMetalHost status

Procedure

1. Get the list of BareMetalHost resources:

   $ oc get bmh -n openshift-machine-api -o yaml

   Note

   You can use baremetalhost as the long form of bmh with oc get command.

2. Get the list of hosts:

   $ oc get bmh -n openshift-machine-api

3. Get the BareMetalHost resource for a specific host:

   $ oc get bmh <host_name> -n openshift-machine-api -o yaml

   Where <host_name> is the name of the host.

   Example output

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     creationTimestamp: "2022-06-16T10:48:33Z"
     finalizers:
     - baremetalhost.metal3.io
     generation: 2
     name: openshift-worker-0
     namespace: openshift-machine-api
     resourceVersion: "30099"
     uid: 1513ae9b-e092-409d-be1b-ad08edeb1271
   spec:
     automatedCleaningMode: metadata
     bmc:
       address: redfish://10.46.61.19:443/redfish/v1/Systems/1
       credentialsName: openshift-worker-0-bmc-secret
       disableCertificateVerification: true
     bootMACAddress: 48:df:37:c7:f7:b0
     bootMode: UEFI
     consumerRef:
       apiVersion: machine.openshift.io/v1beta1
       kind: Machine
       name: ocp-edge-958fk-worker-0-nrfcg
       namespace: openshift-machine-api
     customDeploy:
       method: install_coreos
     online: true
     rootDeviceHints:
       deviceName: /dev/disk/by-id/scsi-<serial_number>
     userData:
       name: worker-user-data-managed
       namespace: openshift-machine-api
   status:
     errorCount: 0
     errorMessage: ""
     goodCredentials:
       credentials:
         name: openshift-worker-0-bmc-secret
         namespace: openshift-machine-api
       credentialsVersion: "16120"
     hardware:
       cpu:
         arch: x86_64
         clockMegahertz: 2300
         count: 64
         flags:
         - 3dnowprefetch
         - abm
         - acpi
         - adx
         - aes
         model: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
       firmware:
         bios:
           date: 10/26/2020
           vendor: HPE
           version: U30
       hostname: openshift-worker-0
       nics:
       - mac: 48:df:37:c7:f7:b3
         model: 0x8086 0x1572
         name: ens1f3
       ramMebibytes: 262144
       storage:
       - hctl: "0:0:0:0"
         model: VK000960GWTTB
         name: /dev/disk/by-id/scsi-<serial_number>
         sizeBytes: 960197124096
         type: SSD
         vendor: ATA
       systemVendor:
         manufacturer: HPE
         productName: ProLiant DL380 Gen10 (868703-B21)
         serialNumber: CZ200606M3
     lastUpdated: "2022-06-16T11:41:42Z"
     operationalStatus: OK
     poweredOn: true
     provisioning:
       ID: 217baa14-cfcf-4196-b764-744e184a3413
       bootMode: UEFI
       customDeploy:
         method: install_coreos
       image:
         url: ""
       raid:
         hardwareRAIDVolumes: null
         softwareRAIDVolumes: []
       rootDeviceHints:
         deviceName: /dev/disk/by-id/scsi-<serial_number>
       state: provisioned
     triedCredentials:
       credentials:
         name: openshift-worker-0-bmc-secret
         namespace: openshift-machine-api
       credentialsVersion: "16120"

Procedure

1. Get the list of nodes:

   $ oc get bmh -n openshift-machine-api

2. Before editing the node’s BareMetalHost resource, detach the node from Ironic by running the following command:

   $ oc annotate baremetalhost <node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached=true' 1

   1

   Replace <node_name> with the name of the node.

3. Edit the BareMetalHost resource by running the following command:

   $ oc edit bmh <node_name> -n openshift-machine-api

4. Reattach the node to Ironic by running the following command:

   $ oc annotate baremetalhost <node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached'-

Procedure

1. If the cleaning process fails and restarts, wait for it to finish. This might take about 5 minutes.
2. If the provisioning status remains in the deleting state, disable the cleaning process by modifying the BareMetalHost resource and setting the automatedCleaningMode field to disabled.

Procedure

1. Create a DataImage resource:

   apiVersion: metal3.io/v1alpha1
   kind: DataImage
   metadata:
     name: <node_name> 1
   spec:
     url: "http://dataimage.example.com/non-bootable.iso" 2

   1

   Specify the name of the node as defined in its BareMetalHost resource.

   2

   Specify the URL and path to the ISO image.

2. Save the DataImage resource to a file by running the following command:

   $ vim <node_name>-dataimage.yaml

3. Apply the DataImage resource by running the following command:

   $ oc apply -f <node_name>-dataimage.yaml -n <node_namespace> 1

   1

   Replace <node_namespace> so that the namespace matches the namespace for the BareMetalHost resource. For example, openshift-machine-api.

4. Reboot the node.

   Note

   To reboot the node, attach the reboot.metal3.io annotation, or reset set the online status in the BareMetalHost resource. A forced reboot of the bare-metal node will change the state of the node to NotReady for awhile. For example, 5 minutes or more.

5. View the DataImage resource by running the following command:

   $ oc get dataimage <node_name> -n openshift-machine-api -o yaml

   Example output

   apiVersion: v1
   items:
   - apiVersion: metal3.io/v1alpha1
     kind: DataImage
     metadata:
       annotations:
         kubectl.kubernetes.io/last-applied-configuration: |
           {"apiVersion":"metal3.io/v1alpha1","kind":"DataImage","metadata":{"annotations":{},"name":"bmh-node-1","namespace":"openshift-machine-api"},"spec":{"url":"http://dataimage.example.com/non-bootable.iso"}}
       creationTimestamp: "2024-06-10T12:00:00Z"
       finalizers:
       - dataimage.metal3.io
       generation: 1
       name: bmh-node-1
       namespace: openshift-machine-api
       ownerReferences:
       - apiVersion: metal3.io/v1alpha1
         blockOwnerDeletion: true
         controller: true
         kind: BareMetalHost
         name: bmh-node-1
         uid: 046cdf8e-0e97-485a-8866-e62d20e0f0b3
       resourceVersion: "21695581"
       uid: c5718f50-44b6-4a22-a6b7-71197e4b7b69
     spec:
       url: http://dataimage.example.com/non-bootable.iso
     status:
       attachedImage:
         url: http://dataimage.example.com/non-bootable.iso
       error:
         count: 0
         message: ""
       lastReconciled: "2024-06-10T12:05:00Z"

1. The HostFirmwareSettings spec.
2. The HostFirmwareSettings status.

Procedure

1. Get the detailed list of HostFirmwareSettings resources:

   $ oc get hfs -n openshift-machine-api -o yaml

   Note

   You can use hostfirmwaresettings as the long form of hfs with the oc get command.

2. Get the list of HostFirmwareSettings resources:

   $ oc get hfs -n openshift-machine-api

3. Get the HostFirmwareSettings resource for a particular host

   $ oc get hfs <host_name> -n openshift-machine-api -o yaml

   Where <host_name> is the name of the host.

Procedure

1. Get the list of HostFirmwareSettings resources:

   $ oc get hfs -n openshift-machine-api

2. Edit a host’s HostFirmwareSettings resource:

   $ oc edit hfs <host_name> -n openshift-machine-api

   Where <host_name> is the name of a provisioned host. The HostFirmwareSettings resource will open in the default editor for your terminal.

3. Add name/value pairs to the spec.settings section:

   Example

   spec:
     settings:
       name: value 1

   1

   Use the FirmwareSchema resource to identify the available settings for the host. You cannot set values that are read-only.

4. Save the changes and exit the editor.

5. Get the host’s machine name:

    $ oc get bmh <host_name> -n openshift-machine name

   Where <host_name> is the name of the host. The machine name appears under the CONSUMER field.

6. Annotate the machine to delete it from the machineset:

   $ oc annotate machine <machine_name> machine.openshift.io/delete-machine=true -n openshift-machine-api

   Where <machine_name> is the name of the machine to delete.

7. Get a list of nodes and count the number of worker nodes:

   $ oc get nodes

8. Get the machineset:

   $ oc get machinesets -n openshift-machine-api

9. Scale the machineset:

   $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n-1>

   Where <machineset_name> is the name of the machineset and <n-1> is the decremented number of worker nodes.

10. When the host enters the Available state, scale up the machineset to make the HostFirmwareSettings resource changes take effect:

    $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n>

    Where <machineset_name> is the name of the machineset and <n> is the number of worker nodes.

Procedure

1. Get a list of HostFirmwareSetting resources:

   $ oc get hfs -n openshift-machine-api

2. Verify that the HostFirmwareSettings resource for a particular host is valid:

   $ oc describe hfs <host_name> -n openshift-machine-api

   Where <host_name> is the name of the host.

   Example output

   Events:
     Type    Reason            Age    From                                    Message
     ----    ------            ----   ----                                    -------
     Normal  ValidationFailed  2m49s  metal3-hostfirmwaresettings-controller  Invalid BIOS setting: Setting ProcTurboMode is invalid, unknown enumeration value - Foo

   Important

   If the response returns ValidationFailed, there is an error in the resource configuration and you must update the values to conform to the FirmwareSchema resource.

Procedure

1. To get a list of FirmwareSchema resource instances, execute the following:

   $ oc get firmwareschema -n openshift-machine-api

2. To get a particular FirmwareSchema instance, execute:

   $ oc get firmwareschema <instance_name> -n openshift-machine-api -o yaml

   Where <instance_name> is the name of the schema instance stated in the HostFirmwareSettings resource (see Table 3).

1. The HostFirmwareComponents spec
2. The HostFirmwareComponents status

Procedure

1. Get the detailed list of HostFirmwareComponents resources:

   $ oc get hostfirmwarecomponents -n openshift-machine-api -o yaml

2. Get the list of HostFirmwareComponents resources:

   $ oc get hostfirmwarecomponents -n openshift-machine-api

3. Get the HostFirmwareComponents resource for a particular host:

   $ oc get hostfirmwarecomponents <host_name> -n openshift-machine-api -o yaml

   Where <host_name> is the name of the host.

   Example output

   ---
   apiVersion: metal3.io/v1alpha1
   kind: HostFirmwareComponents
   metadata:
     creationTimestamp: 2024-04-25T20:32:06Z"
     generation: 1
     name: ostest-master-2
     namespace: openshift-machine-api
     ownerReferences:
     - apiVersion: metal3.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: BareMetalHost
       name: ostest-master-2
       uid: 16022566-7850-4dc8-9e7d-f216211d4195
     resourceVersion: "2437"
     uid: 2038d63f-afc0-4413-8ffe-2f8e098d1f6c
   spec:
     updates: []
   status:
     components:
     - component: bios
       currentVersion: 1.0.0
       initialVersion: 1.0.0
     - component: bmc
       currentVersion: "1.00"
       initialVersion: "1.00"
     conditions:
     - lastTransitionTime: "2024-04-25T20:32:06Z"
       message: ""
       observedGeneration: 1
       reason: OK
       status: "True"
       type: Valid
     - lastTransitionTime: "2024-04-25T20:32:06Z"
       message: ""
       observedGeneration: 1
       reason: OK
       status: "False"
       type: ChangeDetected
     lastUpdated: "2024-04-25T20:32:06Z"
     updates: []

Procedure

1. Get the detailed list of HostFirmwareComponents resources:

   $ oc get hostfirmwarecomponents -n openshift-machine-api -o yaml

2. Edit a host’s HostFirmwareComponents resource:

   $ oc edit <host_name> hostfirmwarecomponents -n openshift-machine-api 1

   1

   Where <host_name> is the name of the host. The HostFirmwareComponents resource will open in the default editor for your terminal.

   Example output

   ---
   apiVersion: metal3.io/v1alpha1
   kind: HostFirmwareComponents
   metadata:
     creationTimestamp: 2024-04-25T20:32:06Z"
     generation: 1
     name: ostest-master-2
     namespace: openshift-machine-api
     ownerReferences:
     - apiVersion: metal3.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: BareMetalHost
       name: ostest-master-2
       uid: 16022566-7850-4dc8-9e7d-f216211d4195
     resourceVersion: "2437"
     uid: 2038d63f-afc0-4413-8ffe-2f8e098d1f6c
   spec:
     updates:
       - name: bios 1
         url: https://myurl.with.firmware.for.bios 2
       - name: bmc 3
         url: https://myurl.with.firmware.for.bmc 4
   status:
     components:
     - component: bios
       currentVersion: 1.0.0
       initialVersion: 1.0.0
     - component: bmc
       currentVersion: "1.00"
       initialVersion: "1.00"
     conditions:
     - lastTransitionTime: "2024-04-25T20:32:06Z"
       message: ""
       observedGeneration: 1
       reason: OK
       status: "True"
       type: Valid
     - lastTransitionTime: "2024-04-25T20:32:06Z"
       message: ""
       observedGeneration: 1
       reason: OK
       status: "False"
       type: ChangeDetected
     lastUpdated: "2024-04-25T20:32:06Z"

   1

   To set a BIOS version, set the name attribute to bios.

   2

   To set a BIOS version, set the url attribute to the URL for the firmware version of the BIOS.

   3

   To set a BMC version, set the name attribute to bmc.

   4

   To set a BMC version, set the url attribute to the URL for the firmware verison of the BMC.

3. Save the changes and exit the editor.

4. Get the host’s machine name:

   $ oc get bmh <host_name> -n openshift-machine name 1

   1

   Where <host_name> is the name of the host. The machine name appears under the CONSUMER field.

5. Annotate the machine to delete it from the machine set:

   $ oc annotate machine <machine_name> machine.openshift.io/delete-machine=true -n openshift-machine-api 1

   1

   Where <machine_name> is the name of the machine to delete.

6. Get a list of nodes and count the number of worker nodes:

   $ oc get nodes

7. Get the machine set:

   $ oc get machinesets -n openshift-machine-api

8. Scale the machine set:

   $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n-1> 1

   1

   Where <machineset_name> is the name of the machine set and <n-1> is the decremented number of worker nodes.

9. When the host enters the Available state, scale up the machine set to make the HostFirmwareComponents resource changes take effect:

   $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n> 1

   1

   Where <machineset_name> is the name of the machine set and <n> is the number of worker nodes.