Search
SupportConsoleDevelopersStart a trialContact

Chapter 5. Scaling a user-provisioned cluster with the Bare Metal Operator

download PDF

After deploying a user-provisioned infrastructure cluster, you can use the Bare Metal Operator (BMO) and other metal3 components to scale bare-metal hosts in the cluster. This approach helps you to scale a user-provisioned cluster in a more automated way.

5.1. About scaling a user-provisioned cluster with the Bare Metal Operator

You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal hosts in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.

5.1.1. Prerequisites for scaling a user-provisioned cluster

  • You installed a user-provisioned infrastructure cluster on bare metal.
  • You have baseboard management controller (BMC) access to the hosts.

5.1.2. Limitations for scaling a user-provisioned cluster

  • You cannot use a provisioning network to scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO).

    • Consequentially, you can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
  • You cannot scale MachineSet objects in user-provisioned infrastructure clusters by using the BMO.

5.2. Configuring a provisioning resource to scale user-provisioned clusters

Create a Provisioning custom resource (CR) to enable Metal platform components on a user-provisioned infrastructure cluster.

Prerequisites

  • You installed a user-provisioned infrastructure cluster on bare metal.

Procedure

  1. Create a Provisioning CR.

    1. Save the following YAML in the provisioning.yaml file: 

      apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  name: provisioning-configuration
spec:
  provisioningNetwork: "Disabled"
  watchAllNamespaces: false
      Note

      OpenShift Container Platform 4.15 does not support enabling a provisioning network when you scale a user-provisioned cluster by using the Bare Metal Operator.

  2. Create the Provisioning CR by running the following command: 

    $ oc create -f provisioning.yaml

    Example output

    provisioning.metal3.io/provisioning-configuration created

Verification

  • Verify that the provisioning service is running by running the following command: 

    $ oc get pods -n openshift-machine-api

    Example output

    NAME                                                  READY   STATUS    RESTARTS        AGE
cluster-autoscaler-operator-678c476f4c-jjdn5          2/2     Running   0               5d21h
cluster-baremetal-operator-6866f7b976-gmvgh           2/2     Running   0               5d21h
control-plane-machine-set-operator-7d8566696c-bh4jz   1/1     Running   0               5d21h
ironic-proxy-64bdw                                    1/1     Running   0               5d21h
ironic-proxy-rbggf                                    1/1     Running   0               5d21h
ironic-proxy-vj54c                                    1/1     Running   0               5d21h
machine-api-controllers-544d6849d5-tgj9l              7/7     Running   1 (5d21h ago)   5d21h
machine-api-operator-5c4ff4b86d-6fjmq                 2/2     Running   0               5d21h
metal3-6d98f84cc8-zn2mx                               5/5     Running   0               5d21h
metal3-image-customization-59d745768d-bhrp7           1/1     Running   0               5d21h

5.3. Provisioning new hosts in a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a user-provisioned cluster by creating a BareMetalHost custom resource (CR).

Note

To provision bare-metal hosts to the cluster by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false.

Prerequisites

  • You created a user-provisioned bare-metal cluster.
  • You have baseboard management controller (BMC) access to the hosts.
  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the bmh.yaml file: 

      ---
apiVersion: v1
kind: Secret
metadata:
  name: worker1-bmc
  namespace: openshift-machine-api
type: Opaque
data:
  username: <base64_of_uid>
  password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: worker1
  namespace: openshift-machine-api
spec:
  bmc:
    address: <protocol>://<bmc_url> 1
    credentialsName: "worker1-bmc"
  bootMACAddress: <nic1_mac_address>
  externallyProvisioned: false 2
  customDeploy:
    method: install_coreos
  online: true
  userData:
    name: worker-user-data-managed
    namespace: openshift-machine-api
      1
      You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2
      You must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false. The default value is false.

  2. Create the bare-metal host object by running the following command: 

    $ oc create -f bmh.yaml

    Example output

    secret/worker1-bmc created
baremetalhost.metal3.io/worker1 created

  3. Approve all certificate signing requests (CSRs).

    1. Verify that the provisioning state of the host is provisioned by running the following command: 

      $ oc get bmh -A

      Example output

      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
openshift-machine-api   controller1   externally provisioned              true             5m25s
openshift-machine-api   worker1       provisioned                         true             4m45s

    2. Get the list of pending CSRs by running the following command: 

      $ oc get csr

      Example output

      NAME        AGE   SIGNERNAME                                    REQUESTOR                                         REQUESTEDDURATION CONDITION
csr-gfm9f   33s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-o
perator:node-bootstrapper   <none>              Pending

    3. Approve the CSR by running the following command: 

      $ oc adm certificate approve <csr_name>

      Example output

      certificatesigningrequest.certificates.k8s.io/<csr_name> approved

Verification

  • Verify that the node is ready by running the following command: 

    $ oc get nodes

    Example output

    NAME        STATUS   ROLES           AGE     VERSION
app1        Ready    worker          47s     v1.24.0+dc5a2fd
controller1 Ready    master,worker   2d22h   v1.24.0+dc5a2fd

5.4. Optional: Managing existing hosts in a user-provisioned cluster by using the BMO

Optionally, you can use the Bare Metal Operator (BMO) to manage existing bare-metal controller hosts in a user-provisioned cluster by creating a BareMetalHost object for the existing host. It is not a requirement to manage existing user-provisioned hosts; however, you can enroll them as externally-provisioned hosts for inventory purposes.

Important

To manage existing hosts by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to true to prevent the BMO from re-provisioning the host.

Prerequisites

  • You created a user-provisioned bare-metal cluster.
  • You have baseboard management controller (BMC) access to the hosts.
  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the controller.yaml file: 

      ---
apiVersion: v1
kind: Secret
metadata:
  name: controller1-bmc
  namespace: openshift-machine-api
type: Opaque
data:
  username: <base64_of_uid>
  password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: controller1
  namespace: openshift-machine-api
spec:
  bmc:
    address: <protocol>://<bmc_url> 1
    credentialsName: "controller1-bmc"
  bootMACAddress: <nic1_mac_address>
  customDeploy:
    method: install_coreos
  externallyProvisioned: true 2
  online: true
  userData:
    name: controller-user-data-managed
    namespace: openshift-machine-api
      1
      You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2
      You must set the value to true to prevent the BMO from re-provisioning the bare-metal controller host.

  2. Create the bare-metal host object by running the following command: 

    $ oc create -f controller.yaml

    Example output

    secret/controller1-bmc created
baremetalhost.metal3.io/controller1 created

Verification

  • Verify that the BMO created the bare-metal host object by running the following command: 

    $ oc get bmh -A

    Example output

    NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
openshift-machine-api   controller1   externally provisioned              true             13s

5.5. Removing hosts from a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user-provisioned cluster.

Prerequisites

  • You created a user-provisioned bare-metal cluster.
  • You have baseboard management controller (BMC) access to the hosts.
  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Cordon and drain the host by running the following command: 

    $ oc adm drain app1 --force --ignore-daemonsets=true

    Example output

    node/app1 cordoned
WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-tvthg, openshift-dns/dns-
default-9q6rz, openshift-dns/node-resolver-zvt42, openshift-image-registry/node-ca-mzxth, openshift-ingress-cana
ry/ingress-canary-qq5lf, openshift-machine-config-operator/machine-config-daemon-v79dm, openshift-monitoring/nod
e-exporter-2vn59, openshift-multus/multus-additional-cni-plugins-wssvj, openshift-multus/multus-fn8tg, openshift
-multus/network-metrics-daemon-5qv55, openshift-network-diagnostics/network-check-target-jqxn2, openshift-ovn-ku
bernetes/ovnkube-node-rsvqg
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766965-258vp
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766950-kg5mk
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766935-stf4s
pod/collect-profiles-27766965-258vp evicted
pod/collect-profiles-27766950-kg5mk evicted
pod/collect-profiles-27766935-stf4s evicted
node/app1 drained

  2. Delete the customDeploy specification from the BareMetalHost CR.

    1. Edit the BareMetalHost CR for the host by running the following command: 

      $ oc edit bmh -n openshift-machine-api <host_name>

    2. Delete the lines spec.customDeploy and spec.customDeploy.method: 

      ...
  customDeploy:
    method: install_coreos

    3. Verify that the provisioning state of the host changes to deprovisioning by running the following command: 

      $ oc get bmh -A

      Example output

      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
openshift-machine-api   controller1   externally provisioned              true             58m
openshift-machine-api   worker1       deprovisioning                      true             57m

  3. Delete the node by running the following command: 

    $ oc delete node <node_name>

Verification

  • Verify the node is deleted by running the following command: 

    $ oc get nodes

    Example output

    NAME          STATUS   ROLES           AGE     VERSION
controller1   Ready    master,worker   2d23h   v1.24.0+dc5a2fd

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.