Chapter 5. Scaling a user-provisioned cluster with the Bare Metal Operator

Procedure

1. Create a Provisioning CR.

   1. Save the following YAML in the provisioning.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: Provisioning
      metadata:
        name: provisioning-configuration
      spec:
        provisioningNetwork: "Disabled"
        watchAllNamespaces: false

      Note

      OpenShift Container Platform 4.15 does not support enabling a provisioning network when you scale a user-provisioned cluster by using the Bare Metal Operator.

2. Create the Provisioning CR by running the following command:

   $ oc create -f provisioning.yaml

   Example output

   provisioning.metal3.io/provisioning-configuration created

Procedure

1. Create a configuration file for the bare-metal node. Depending if you use either a static configuration or a DHCP server, choose one of the following example bmh.yaml files and configure it to your needs by replacing values in the YAML to match your environment:

   • To deploy with a static configuration, create the following bmh.yaml file:

     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-network-config-secret 1
       namespace: openshift-machine-api
     type: Opaque
     stringData:
       nmstate: | 2
         interfaces: 3
         - name: <nic1_name> 4
           type: ethernet
           state: up
           ipv4:
             address:
             - ip: <ip_address> 5
               prefix-length: 24
             enabled: true
         dns-resolver:
           config:
             server:
             - <dns_ip_address> 6
         routes:
           config:
           - destination: 0.0.0.0/0
             next-hop-address: <next_hop_ip_address> 7
             next-hop-interface: <next_hop_nic1_name> 8
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 9
       password: <base64_of_pwd>
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num>
       namespace: openshift-machine-api
     spec:
       online: true
       bootMACAddress: <nic1_mac_address> 10
       bmc:
         address: <protocol>://<bmc_url> 11
         credentialsName: openshift-worker-<num>-bmc-secret
         disableCertificateVerification: false
       customDeploy:
         method: install_coreos
       userData:
         name: worker-user-data-managed
         namespace: openshift-machine-api
       rootDeviceHints:
         deviceName: <root_device_hint> 12
       preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret

     1

     Replace all instances of <num> with a unique compute node number for the bare-metal nodes in the name, credentialsName, and preprovisioningNetworkDataName fields.

     2

     Add the NMState YAML syntax to configure the host interfaces. To configure the network interface for a newly created node, specify the name of the secret that has the network configuration. Follow the nmstate syntax to define the network configuration for your node. See "Preparing the bare-metal node" for details on configuring NMState syntax.

     3

     Optional: If you have configured the network interface with nmstate, and you want to disable an interface, set state: up with the IP addresses set to enabled: false.

     4

     Replace <nic1_name> with the name of the bare-metal node’s first network interface controller (NIC).

     5

     Replace <ip_address> with the IP address of the bare-metal node’s NIC.

     6

     Replace <dns_ip_address> with the IP address of the bare-metal node’s DNS resolver.

     7

     Replace <next_hop_ip_address> with the IP address of the bare-metal node’s external gateway.

     8

     Replace <next_hop_nic1_name> with the name of the bare-metal node’s external gateway.

     9

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     10

     Replace <nic1_mac_address> with the MAC address of the bare-metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     11

     Replace <protocol> with the BMC protocol, such as IPMI, Redfish, or others. Replace <bmc_url> with the URL of the bare-metal node’s baseboard management controller.

     12

     Optional: Replace <root_device_hint> with a device path when specifying a root device hint. See "Root device hints" for additional details.

   • When configuring the network interface with a static configuration by using nmstate, set state: up with the IP addresses set to enabled: false:

     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-network-config-secret
       namespace: openshift-machine-api
      # ...
     interfaces:
       - name: <nic_name>
         type: ethernet
         state: up
         ipv4:
           enabled: false
         ipv6:
           enabled: false
     # ...

   • To deploy with a DHCP configuration, create the following bmh.yaml file:

     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret 1
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 2
       password: <base64_of_pwd>
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num>
       namespace: openshift-machine-api
     spec:
       online: true
       bootMACAddress: <nic1_mac_address> 3
       bmc:
         address: <protocol>://<bmc_url> 4
         credentialsName: openshift-worker-<num>-bmc
         disableCertificateVerification: false
       customDeploy:
         method: install_coreos
       userData:
         name: worker-user-data-managed
         namespace: openshift-machine-api
       rootDeviceHints:
         deviceName: <root_device_hint> 5

     1

     Replace <num> with a unique compute node number for the bare-metal nodes in the name and credentialsName fields.

     2

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     3

     Replace <nic1_mac_address> with the MAC address of the bare-metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     4

     Replace <protocol> with the BMC protocol, such as IPMI, Redfish, or others. Replace <bmc_url> with the URL of the bare-metal node’s baseboard management controller.

     5

     Optional: Replace <root_device_hint> with a device path when specifying a root device hint. See "Root device hints" for additional details.

     Important

     If the MAC address of an existing bare-metal node matches the MAC address of the bare-metal host that you are attempting to provision, then the installation will fail. If the host enrollment, inspection, cleaning, or other steps fail, the Bare Metal Operator retries the installation continuously. See "Diagnosing a duplicate MAC address when provisioning a new host in the cluster" for additional details.

2. Create the bare-metal node by running the following command:

   $ oc create -f bmh.yaml

   Example output

   secret/openshift-worker-<num>-network-config-secret created
   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

3. Inspect the bare-metal node by running the following command:

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   where:

   <num>

   Specifies the compute node number.

   Example output

   NAME                    STATE       CONSUMER   ONLINE   ERROR
   openshift-worker-<num>  provisioned true

4. Approve all certificate signing requests (CSRs).

   1. Get the list of pending CSRs by running the following command:

      $ oc get csr

      Example output

      NAME        AGE   SIGNERNAME                                    REQUESTOR                                         REQUESTEDDURATION CONDITION
      csr-gfm9f   33s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-o
      perator:node-bootstrapper   <none>              Pending

   2. Approve the CSR by running the following command:

      $ oc adm certificate approve <csr_name>

      Example output

      certificatesigningrequest.certificates.k8s.io/<csr_name> approved

Procedure

1. Create the Secret CR and the BareMetalHost CR.

   1. Save the following YAML in the controller.yaml file:

      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: controller1-bmc
        namespace: openshift-machine-api
      type: Opaque
      data:
        username: <base64_of_uid>
        password: <base64_of_pwd>
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: controller1
        namespace: openshift-machine-api
      spec:
        bmc:
          address: <protocol>://<bmc_url> 1
          credentialsName: "controller1-bmc"
        bootMACAddress: <nic1_mac_address>
        customDeploy:
          method: install_coreos
        externallyProvisioned: true 2
        online: true
        userData:
          name: controller-user-data-managed
          namespace: openshift-machine-api

      1

      You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.

      2

      You must set the value to true to prevent the BMO from re-provisioning the bare-metal controller host.

2. Create the bare-metal host object by running the following command:

   $ oc create -f controller.yaml

   Example output

   secret/controller1-bmc created
   baremetalhost.metal3.io/controller1 created

Procedure

1. Cordon and drain the node by running the following command:

   $ oc adm drain app1 --force --ignore-daemonsets=true

   Example output

   node/app1 cordoned
   WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-tvthg, openshift-dns/dns-
   default-9q6rz, openshift-dns/node-resolver-zvt42, openshift-image-registry/node-ca-mzxth, openshift-ingress-cana
   ry/ingress-canary-qq5lf, openshift-machine-config-operator/machine-config-daemon-v79dm, openshift-monitoring/nod
   e-exporter-2vn59, openshift-multus/multus-additional-cni-plugins-wssvj, openshift-multus/multus-fn8tg, openshift
   -multus/network-metrics-daemon-5qv55, openshift-network-diagnostics/network-check-target-jqxn2, openshift-ovn-ku
   bernetes/ovnkube-node-rsvqg
   evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766965-258vp
   evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766950-kg5mk
   evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766935-stf4s
   pod/collect-profiles-27766965-258vp evicted
   pod/collect-profiles-27766950-kg5mk evicted
   pod/collect-profiles-27766935-stf4s evicted
   node/app1 drained

2. Delete the customDeploy specification from the BareMetalHost CR.

   1. Edit the BareMetalHost CR for the host by running the following command:

      $ oc edit bmh -n openshift-machine-api <host_name>

   2. Delete the lines spec.customDeploy and spec.customDeploy.method:

      ...
        customDeploy:
          method: install_coreos

   3. Verify that the provisioning state of the host changes to deprovisioning by running the following command:

      $ oc get bmh -A

      Example output

      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
      openshift-machine-api   controller1   externally provisioned              true             58m
      openshift-machine-api   worker1       deprovisioning                      true             57m

3. Delete the host by running the following command when the BareMetalHost state changes to available:

   $ oc delete bmh -n openshift-machine-api <bmh_name>

   Note

   You can run this step without having to edit the BareMetalHost CR. It might take some time for the BareMetalHost state to change from deprovisioning to available.

4. Delete the node by running the following command:

   $ oc delete node <node_name>