Chapter 7. User-provisioned infrastructure

7.1. Adding RHEL compute machines to an OpenShift Container Platform cluster

In OpenShift Container Platform, you can add Red Hat Enterprise Linux (RHEL) compute, or worker, machines to a user-provisioned infrastructure cluster. You can use RHEL as the operating system on only compute machines.

7.1.1. About adding RHEL compute nodes to a cluster

In OpenShift Container Platform 4.4, you have the option of using Red Hat Enterprise Linux (RHEL) machines as compute machines, which are also known as worker machines, in your cluster if you use a user-provisioned infrastructure installation. You must use Red Hat Enterprise Linux CoreOS (RHCOS) machines for the control plane, or master, machines in your cluster.

As with all installations that use user-provisioned infrastructure, if you choose to use RHEL compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks.

Important

Because removing OpenShift Container Platform from a machine in the cluster requires destroying the operating system, you must use dedicated hardware for any RHEL machines that you add to the cluster.

Important

Swap memory is disabled on all RHEL machines that you add to your OpenShift Container Platform cluster. You cannot enable swap memory on these machines.

You must add any RHEL compute machines to the cluster after you initialize the control plane.

7.1.2. System requirements for RHEL compute nodes

The Red Hat Enterprise Linux (RHEL) compute machine hosts, which are also known as worker machine hosts, in your OpenShift Container Platform environment must meet the following minimum hardware specifications and system-level requirements.

You must have an active OpenShift Container Platform subscription on your Red Hat account. If you do not, contact your sales representative for more information.
Production environments must provide compute machines to support your expected workloads. As a cluster administrator, you must calculate the expected workload and add about 10 percent for overhead. For production environments, allocate enough resources so that a node host failure does not affect your maximum capacity.
Each system must meet the following hardware requirements:
- Physical or virtual system, or an instance running on a public or private IaaS.
- Base OS: RHEL 7.6-7.8 with "Minimal" installation option.
  Important
  Only RHEL 7.6-7.8 is supported in OpenShift Container Platform 4.4. You must not upgrade your compute machines to RHEL 8.
- If you deployed OpenShift Container Platform in FIPS mode, you must enable FIPS on the RHEL machine before you boot it. See Enabling FIPS Mode in the RHEL 7 documentation.
- NetworkManager 1.0 or later.
- 1 vCPU.
- Minimum 8 GB RAM.
- Minimum 15 GB hard disk space for the file system containing /var/.
- Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.
- Minimum 1 GB hard disk space for the file system containing the system’s temporary directory. The system’s temporary directory is determined according to the rules defined in the tempfile module in Python’s standard library.
Each system must meet any additional requirements for your system provider. For example, if you installed your cluster on VMware vSphere, your disks must be configured according to its storage guidelines and the disk.enableUUID=true attribute must be set.
Each system must be able to access the cluster’s API endpoints by using DNS-resolvable host names. Any network security access control that is in place must allow the system access to the cluster’s API service endpoints.

7.1.2.1. Certificate signing requests management

Because your cluster has limited access to automatic machine management when you use infrastructure that you provision, you must provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The machine-approver cannot guarantee the validity of a serving certificate that is requested by using kubelet credentials because it cannot confirm that the correct machine issued the request. You must determine and implement a method of verifying the validity of the kubelet serving certificate requests and approving them.

7.1.3. Preparing the machine to run the playbook

Before you can add compute machines that use Red Hat Enterprise Linux as the operating system to an OpenShift Container Platform 4.4 cluster, you must prepare a machine to run the playbook from. This machine is not part of the cluster but must be able to access it.

Prerequisites

Install the OpenShift CLI (oc) on the machine that you run the playbook on.
Log in as a user with cluster-admin permission.

Procedure

Ensure that the kubeconfig file for the cluster and the installation program that you used to install the cluster are on the machine. One way to accomplish this is to use the same machine that you used to install the cluster.
Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.
Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.
Important
If you use SSH key-based authentication, you must manage the key with an SSH agent.
If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift subscription to it:
1. Register the machine with RHSM:
```
# subscription-manager register --username=<user_name> --password=<password>
```
2. Pull the latest subscription data from RHSM:
```
# subscription-manager refresh
```
3. List the available subscriptions:
```
# subscription-manager list --available --matches '*OpenShift*'
```
4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:
```
# subscription-manager attach --pool=<pool_id>
```

Enable the repositories required by OpenShift Container Platform 4.4:

# subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ansible-2.9-rpms" \
    --enable="rhel-7-server-ose-4.4-rpms"

Install the required packages, including openshift-ansible:
```
# yum install openshift-ansible openshift-clients jq
```
The openshift-ansible package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients provides the oc CLI, and the jq package improves the display of JSON output on your command line.

7.1.4. Preparing a RHEL compute node

Before you add a Red Hat Enterprise Linux (RHEL) machine to your OpenShift Container Platform cluster, you must register each host with Red Hat Subscription Manager (RHSM), attach an active OpenShift Container Platform subscription, and enable the required repositories.

On each host, register with RHSM:

# subscription-manager register --username=<user_name> --password=<password>

Pull the latest subscription data from RHSM:
```
# subscription-manager refresh
```

List the available subscriptions:

# subscription-manager list --available --matches '*OpenShift*'

In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:
```
# subscription-manager attach --pool=<pool_id>
```
Disable all yum repositories:
1. Disable all the enabled RHSM repositories:
```
# subscription-manager repos --disable="*"
```
2. List the remaining yum repositories and note their names under repo id, if any:
```
# yum repolist
```
3. Use yum-config-manager to disable the remaining yum repositories:
```
# yum-config-manager --disable <repo_id>
```
  Alternatively, disable all repositories:
```
 yum-config-manager --disable \*
```
  Note that this might take a few minutes if you have a large number of available repositories

Enable only the repositories required by OpenShift Container Platform 4.4:

# subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-4.4-rpms"

Stop and disable firewalld on the host:
```
# systemctl disable --now firewalld.service
```
Note
You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

7.1.5. Adding a RHEL compute machine to your cluster

You can add compute machines that use Red Hat Enterprise Linux as the operating system to an OpenShift Container Platform 4.4 cluster.

Prerequisites

You installed the required packages and performed the necessary configuration on the machine that you run the playbook on.
You prepared the RHEL hosts for installation.

Procedure

Perform the following steps on the machine that you prepared to run the playbook:

Create an Ansible inventory file that is named /<path>/inventory/hosts that defines your compute machine hosts and required variables:
```
[all:vars]
ansible_user=root 1
#ansible_become=True 2

openshift_kubeconfig_path="~/.kube/config" 3

[new_workers] 4
mycluster-rhel7-0.example.com
mycluster-rhel7-1.example.com
```
1
Specify the user name that runs the Ansible tasks on the remote compute machines.
2
If you do not specify root for the ansible_user, you must set ansible_become to True and assign the user sudo permissions.
3
Specify the path and file name of the kubeconfig file for your cluster.
4
List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the host name that the cluster uses to access the machine, so set the correct public or private name to access the machine.

Run the playbook:

$ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

1: For <path>, specify the path to the Ansible inventory file that you created.

7.1.6. Approving the certificate signing requests for your machines

When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself. The client requests must be approved first, followed by the server requests.

Prerequisites

You added machines to your cluster.

Procedure

Confirm that the cluster recognizes the machines:

# oc get nodes

NAME                    STATUS   ROLES    AGE   VERSION
master-01.example.com   Ready    master   40d   v1.17.1
master-02.example.com   Ready    master   40d   v1.17.1
master-03.example.com   Ready    master   40d   v1.17.1
worker-01.example.com   Ready    worker   40d   v1.17.1
worker-02.example.com   Ready    worker   40d   v1.17.1

The output lists all of the machines that you created.

Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

$ oc get csr

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...

In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:
Note
Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

$ oc get csr

Example output

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
...

If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

$ oc get nodes

Example output

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  73m  v1.20.0
master-1  Ready     master  73m  v1.20.0
master-2  Ready     master  74m  v1.20.0
worker-0  Ready     worker  11m  v1.20.0
worker-1  Ready     worker  11m  v1.20.0

Note

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

For more information on CSRs, see Certificate Signing Requests.

7.1.7. Required parameters for the Ansible hosts file

You must define the following parameters in the Ansible hosts file before you add Red Hat Enterprise Linux (RHEL) compute machines to your cluster.

Paramter	Description	Values
`ansible_user`	The SSH user that allows SSH-based authentication without requiring a password. If you use SSH key-based authentication, then you must manage the key with an SSH agent.	A user name on the system. The default value is `root`.
`ansible_become`	If the values of `ansible_user` is not root, you must set `ansible_become` to `True`, and the user that you specify as the `ansible_user` must be configured for passwordless sudo access.	`True`. If the value is not `True`, do not specify and define this parameter.
`openshift_kubeconfig_path`	Specifies a path and file name to a local directory that contains the `kubeconfig` file for your cluster.	The path and name of the configuration file.

7.1.7.1. Optional: Removing RHCOS compute machines from a cluster

After you add the Red Hat Enterprise Linux (RHEL) compute machines to your cluster, you can optionally remove the Red Hat Enterprise Linux CoreOS (RHCOS) compute machines to free up resources.

Prerequisites

You have added RHEL compute machines to your cluster.

Procedure

View the list of machines and record the node names of the RHCOS compute machines:
```
$ oc get nodes -o wide
```
For each RHCOS compute machine, delete the node:
1. Mark the node as unschedulable by running the oc adm cordon command:
```
$ oc adm cordon <node_name> 1
```
  1
  Specify the node name of one of the RHCOS compute machines.
2. Drain all the pods from the node:
```
$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets 1
```
  1
  Specify the node name of the RHCOS compute machine that you isolated.
3. Delete the node:
```
$ oc delete nodes <node_name> 1
```
  1
  Specify the node name of the RHCOS compute machine that you drained.
Review the list of compute machines to ensure that only the RHEL nodes remain:
```
$ oc get nodes -o wide
```
Remove the RHCOS machines from the load balancer for your cluster’s compute machines. You can delete the virtual machines or reimage the physical hardware for the RHCOS compute machines.

7.2. Adding more RHEL compute machines to an OpenShift Container Platform cluster

If your OpenShift Container Platform cluster already includes Red Hat Enterprise Linux (RHEL) compute machines, which are also known as worker machines, you can add more RHEL compute machines to it.

7.2.1. About adding RHEL compute nodes to a cluster

Important

Swap memory is disabled on all RHEL machines that you add to your OpenShift Container Platform cluster. You cannot enable swap memory on these machines.

You must add any RHEL compute machines to the cluster after you initialize the control plane.

7.2.2. System requirements for RHEL compute nodes

You must have an active OpenShift Container Platform subscription on your Red Hat account. If you do not, contact your sales representative for more information.
Production environments must provide compute machines to support your expected workloads. As a cluster administrator, you must calculate the expected workload and add about 10 percent for overhead. For production environments, allocate enough resources so that a node host failure does not affect your maximum capacity.
Each system must meet the following hardware requirements:
- Physical or virtual system, or an instance running on a public or private IaaS.
- Base OS: RHEL 7.6-7.8 with "Minimal" installation option.
  Important
  Only RHEL 7.6-7.8 is supported in OpenShift Container Platform 4.4. You must not upgrade your compute machines to RHEL 8.
- If you deployed OpenShift Container Platform in FIPS mode, you must enable FIPS on the RHEL machine before you boot it. See Enabling FIPS Mode in the RHEL 7 documentation.
- NetworkManager 1.0 or later.
- 1 vCPU.
- Minimum 8 GB RAM.
- Minimum 15 GB hard disk space for the file system containing /var/.
- Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.
- Minimum 1 GB hard disk space for the file system containing the system’s temporary directory. The system’s temporary directory is determined according to the rules defined in the tempfile module in Python’s standard library.
Each system must meet any additional requirements for your system provider. For example, if you installed your cluster on VMware vSphere, your disks must be configured according to its storage guidelines and the disk.enableUUID=true attribute must be set.
Each system must be able to access the cluster’s API endpoints by using DNS-resolvable host names. Any network security access control that is in place must allow the system access to the cluster’s API service endpoints.

7.2.2.1. Certificate signing requests management

7.2.3. Preparing a RHEL compute node

On each host, register with RHSM:

# subscription-manager register --username=<user_name> --password=<password>

Pull the latest subscription data from RHSM:
```
# subscription-manager refresh
```

List the available subscriptions:

# subscription-manager list --available --matches '*OpenShift*'

In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:
```
# subscription-manager attach --pool=<pool_id>
```
Disable all yum repositories:
1. Disable all the enabled RHSM repositories:
```
# subscription-manager repos --disable="*"
```
2. List the remaining yum repositories and note their names under repo id, if any:
```
# yum repolist
```
3. Use yum-config-manager to disable the remaining yum repositories:
```
# yum-config-manager --disable <repo_id>
```
  Alternatively, disable all repositories:
```
 yum-config-manager --disable \*
```
  Note that this might take a few minutes if you have a large number of available repositories

Enable only the repositories required by OpenShift Container Platform 4.4:

# subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-4.4-rpms"

Stop and disable firewalld on the host:
```
# systemctl disable --now firewalld.service
```
Note
You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

7.2.4. Adding more RHEL compute machines to your cluster

You can add more compute machines that use Red Hat Enterprise Linux (RHEL) as the operating system to an OpenShift Container Platform 4.4 cluster.

Prerequisites

Your OpenShift Container Platform cluster already contains RHEL compute nodes.
The hosts file that you used to add the first RHEL compute machines to your cluster is on the machine that you use the run the playbook.
The machine that you run the playbook on must be able to access all of the RHEL hosts. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.
The kubeconfig file for the cluster and the installation program that you used to install the cluster are on the machine that you use the run the playbook.
You must prepare the RHEL hosts for installation.
Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.
If you use SSH key-based authentication, you must manage the key with an SSH agent.
Install the OpenShift CLI (oc) on the machine that you run the playbook on.

Procedure

Open the Ansible inventory file at /<path>/inventory/hosts that defines your compute machine hosts and required variables.
Rename the [new_workers] section of the file to [workers].
Add a [new_workers] section to the file and define the fully-qualified domain names for each new host. The file resembles the following example:
```
[all:vars]
ansible_user=root
#ansible_become=True

openshift_kubeconfig_path="~/.kube/config"

[workers]
mycluster-rhel7-0.example.com
mycluster-rhel7-1.example.com

[new_workers]
mycluster-rhel7-2.example.com
mycluster-rhel7-3.example.com
```
In this example, the mycluster-rhel7-0.example.com and mycluster-rhel7-1.example.com machines are in the cluster and you add the mycluster-rhel7-2.example.com and mycluster-rhel7-3.example.com machines.

Run the scaleup playbook:

$ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

1: For <path>, specify the path to the Ansible inventory file that you created.

7.2.5. Approving the certificate signing requests for your machines

Prerequisites

You added machines to your cluster.

Procedure

Confirm that the cluster recognizes the machines:

# oc get nodes

NAME                    STATUS   ROLES    AGE   VERSION
master-01.example.com   Ready    master   40d   v1.17.1
master-02.example.com   Ready    master   40d   v1.17.1
master-03.example.com   Ready    master   40d   v1.17.1
worker-01.example.com   Ready    worker   40d   v1.17.1
worker-02.example.com   Ready    worker   40d   v1.17.1

The output lists all of the machines that you created.

Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

$ oc get csr

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...

In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:
Note
Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

$ oc get csr

Example output

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
...

If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

$ oc get nodes

Example output

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  73m  v1.20.0
master-1  Ready     master  73m  v1.20.0
master-2  Ready     master  74m  v1.20.0
worker-0  Ready     worker  11m  v1.20.0
worker-1  Ready     worker  11m  v1.20.0

Note

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

For more information on CSRs, see Certificate Signing Requests.

7.2.6. Required parameters for the Ansible hosts file

You must define the following parameters in the Ansible hosts file before you add Red Hat Enterprise Linux (RHEL) compute machines to your cluster.

Paramter	Description	Values
`ansible_user`	The SSH user that allows SSH-based authentication without requiring a password. If you use SSH key-based authentication, then you must manage the key with an SSH agent.	A user name on the system. The default value is `root`.
`ansible_become`	If the values of `ansible_user` is not root, you must set `ansible_become` to `True`, and the user that you specify as the `ansible_user` must be configured for passwordless sudo access.	`True`. If the value is not `True`, do not specify and define this parameter.
`openshift_kubeconfig_path`	Specifies a path and file name to a local directory that contains the `kubeconfig` file for your cluster.	The path and name of the configuration file.

7.3. Adding compute machines to vSphere

You can add more compute machines to your OpenShift Container Platform cluster on VMware vSphere.

7.3.1. Prerequisites

You installed a cluster on vSphere.

7.3.2. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines in vSphere

You can create more compute machines for your cluster that uses user-provisioned infrastructure on VMware vSphere.

Prerequisites

Obtain the base64-encoded Ignition file for your compute machines.
You have access to the vSphere template that you created for your cluster.

Procedure

After the template deploys, deploy a VM for a machine in the cluster.
1. Right-click the template’s name and click Clone Clone to Virtual Machine.
2. On the Select a name and folder tab, specify a name for the VM. You might include the machine type in the name, such as compute-1.
3. On the Select a name and folder tab, select the name of the folder that you created for the cluster.
4. On the Select a compute resource tab, select the name of a host in your datacenter.
5. Optional: On the Select storage tab, customize the storage options.
6. On the Select clone options, select Customize this virtual machine’s hardware.
7. On the Customize hardware tab, click VM Options Advanced.
  - From the Latency Sensitivity list, select High.
  - Click Edit Configuration, and on the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:
    guestinfo.ignition.config.data: Paste the contents of the base64-encoded compute Ignition config file for this machine type.
    guestinfo.ignition.config.data.encoding: Specify base64.
    disk.EnableUUID: Specify TRUE.
8. In the Virtual Hardware panel of the Customize hardware tab, modify the specified values as required. Ensure that the amount of RAM, CPU, and disk storage meets the minimum requirements for the machine type. Also, make sure to select the correct network under Add network adapter if there are multiple networks available.
9. Complete the configuration and power on the VM.
Continue to create more compute machines for your cluster.

7.3.3. Approving the certificate signing requests for your machines

Prerequisites

You added machines to your cluster.

Procedure

Confirm that the cluster recognizes the machines:

# oc get nodes

NAME                    STATUS   ROLES    AGE   VERSION
master-01.example.com   Ready    master   40d   v1.17.1
master-02.example.com   Ready    master   40d   v1.17.1
master-03.example.com   Ready    master   40d   v1.17.1
worker-01.example.com   Ready    worker   40d   v1.17.1
worker-02.example.com   Ready    worker   40d   v1.17.1

The output lists all of the machines that you created.

Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

$ oc get csr

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...

In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:
Note
Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

$ oc get csr

Example output

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
...

If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

$ oc get nodes

Example output

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  73m  v1.20.0
master-1  Ready     master  73m  v1.20.0
master-2  Ready     master  74m  v1.20.0
worker-0  Ready     worker  11m  v1.20.0
worker-1  Ready     worker  11m  v1.20.0

Note

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

For more information on CSRs, see Certificate Signing Requests.

7.4. Adding compute machines to bare metal

You can add more compute machines to your OpenShift Container Platform cluster on bare metal.

7.4.1. Prerequisites

You installed a cluster on bare metal.
You have installation media and Red Hat Enterprise Linux CoreOS (RHCOS) images that you used to create your cluster. If you do not have these files, you must obtain them by following the instructions in the installation procedure.

7.4.2. Creating Red Hat Enterprise Linux CoreOS (RHCOS) machines

Before you add more compute machines to a cluster that you installed on bare metal infrastructure, you must create RHCOS machines for it to use. Follow either the steps to use an ISO image or network PXE booting to create the machines.

7.4.2.1. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines using an ISO image

You can create more compute machines for your bare metal cluster by using an ISO image to create the machines.

Prerequisites

Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.
Obtain the URL of the BIOS or UEFI RHCOS image file that you uploaded to your HTTP server during cluster installation.

Procedure

Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:
- Burn the ISO image to a disk and boot it directly.
- Use ISO redirection with a LOM interface.
After the instance boots, press the TAB or E key to edit the kernel command line.
Add the parameters to the kernel command line:
```
coreos.inst=yes
coreos.inst.install_dev=sda 1
coreos.inst.image_url=<bare_metal_image_URL> 2
coreos.inst.ignition_url=http://example.com/worker.ign 3
```
1
Specify the block device of the system to install to.
2
Specify the URL of the UEFI or BIOS image that you uploaded to your server.
3
Specify the URL of the compute Ignition config file.
Press Enter to complete the installation. After RHCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.
Continue to create more compute machines for your cluster.

7.4.2.2. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines by PXE or iPXE booting

You can create more compute machines for your bare metal cluster by using PXE or iPXE booting.

Prerequisites

Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.
Obtain the URLs of the RHCOS ISO image, compressed metal BIOS, kernel, and initramfs files that you uploaded to your HTTP server during cluster installation.
You have access to the PXE booting infrastructure that you used to create the machines for your OpenShift Container Platform cluster during installation. The machines must boot from their local disks after RHCOS is installed on them.
If you use UEFI, you have access to the grub.conf file that you modified during OpenShift Container Platform installation.

Procedure

Confirm that your PXE or iPXE installation for the RHCOS images is correct.
- For PXE:
```
DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL http://<HTTP_server>/rhcos-<version>-installer-kernel-<architecture> 1
    APPEND ip=dhcp rd.neednet=1 initrd=http://<HTTP_server>/rhcos-<version>-installer-initramfs.<architecture>.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://<HTTP_server>/rhcos-<version>-metal.<architecture>.raw.gz coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 2 3
```
  1
  Specify the location of the kernel file that you uploaded to your HTTP server.
  2
  If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
  3
  Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the initramfs file, the coreos.inst.image_url parameter value is the location of the compressed metal RAW image, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.
- For iPXE:
```
kernel http://<HTTP_server>/rhcos-<version>-installer-kernel-<architecture> ip=dhcp rd.neednet=1 initrd=http://<HTTP_server>/rhcos-<version>-installer-initramfs.<architecture>.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://<HTTP_server>/rhcos-<version>-metal.<arhcitectutre>.raw.gz coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
initrd http://<HTTP_server>/rhcos-<version>-installer-initramfs.<architecture>.img 3
boot
```
  1
  Specify locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd parameter value is the location of the initramfs file, the coreos.inst.image_url parameter value is the location of the compressed metal RAW image, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.
  2
  If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
  3
  Specify the location of the initramfs file that you uploaded to your HTTP server.
Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

7.4.3. Approving the certificate signing requests for your machines

Prerequisites

You added machines to your cluster.

Procedure

Confirm that the cluster recognizes the machines:

# oc get nodes

NAME                    STATUS   ROLES    AGE   VERSION
master-01.example.com   Ready    master   40d   v1.17.1
master-02.example.com   Ready    master   40d   v1.17.1
master-03.example.com   Ready    master   40d   v1.17.1
worker-01.example.com   Ready    worker   40d   v1.17.1
worker-02.example.com   Ready    worker   40d   v1.17.1

The output lists all of the machines that you created.

Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

$ oc get csr

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...

In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:
Note
Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

$ oc get csr

Example output

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
...

If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:
- To approve them individually, run the following command for each valid CSR:
```
$ oc adm certificate approve <csr_name> 1
```
  1
  <csr_name> is the name of a CSR from the list of current CSRs.
- To approve all pending CSRs, run the following command:
```
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
```

After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

$ oc get nodes

Example output

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  73m  v1.20.0
master-1  Ready     master  73m  v1.20.0
master-2  Ready     master  74m  v1.20.0
worker-0  Ready     worker  11m  v1.20.0
worker-1  Ready     worker  11m  v1.20.0

Note

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

For more information on CSRs, see Certificate Signing Requests.

Chapter 7. User-provisioned infrastructure

7.1. Adding RHEL compute machines to an OpenShift Container Platform cluster

7.1.1. About adding RHEL compute nodes to a cluster

7.1.2. System requirements for RHEL compute nodes

7.1.2.1. Certificate signing requests management

7.1.3. Preparing the machine to run the playbook

7.1.4. Preparing a RHEL compute node

7.1.5. Adding a RHEL compute machine to your cluster

7.1.6. Approving the certificate signing requests for your machines

7.1.7. Required parameters for the Ansible hosts file

7.1.7.1. Optional: Removing RHCOS compute machines from a cluster

7.2. Adding more RHEL compute machines to an OpenShift Container Platform cluster

7.2.1. About adding RHEL compute nodes to a cluster

7.2.2. System requirements for RHEL compute nodes

7.2.2.1. Certificate signing requests management

7.2.3. Preparing a RHEL compute node

7.2.4. Adding more RHEL compute machines to your cluster

7.2.5. Approving the certificate signing requests for your machines

7.2.6. Required parameters for the Ansible hosts file

7.3. Adding compute machines to vSphere

7.3.1. Prerequisites

7.3.2. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines in vSphere

7.3.3. Approving the certificate signing requests for your machines

7.4. Adding compute machines to bare metal

7.4.1. Prerequisites

7.4.2. Creating Red Hat Enterprise Linux CoreOS (RHCOS) machines

7.4.2.1. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines using an ISO image

7.4.2.2. Creating more Red Hat Enterprise Linux CoreOS (RHCOS) machines by PXE or iPXE booting

7.4.3. Approving the certificate signing requests for your machines

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links