Chapter 4. Manage nodes using machine pools

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Machine pools tab, click Add machine pool.
3. Add a Machine pool name.

4. Select a Compute node instance type from the drop-down menu. The instance type defines the vCPU and memory allocation for each compute node in the machine pool.

   Note

   You cannot change the instance type for a machine pool after the pool is created.

5. Optional: Configure autoscaling for the machine pool:

   1. Select Enable autoscaling to automatically scale the number of machines in your machine pool to meet the deployment needs.

   2. Set the minimum and maximum node count limits for autoscaling. The cluster autoscaler does not reduce or increase the machine pool node count beyond the limits that you specify.

      • If you deployed your cluster using a single availability zone, set the Minimum and maximum node count. This defines the minimum and maximum compute node limits in the availability zone.

      • If you deployed your cluster using multiple availability zones, set the Minimum nodes per zone and Maximum nodes per zone. This defines the minimum and maximum compute node limits per zone.

        Note

        Alternatively, you can set your autoscaling preferences for the machine pool after the machine pool is created.

6. If you did not enable autoscaling, select a compute node count:

   • If you deployed your cluster using a single availability zone, select a Compute node count from the drop-down menu. This defines the number of compute nodes to provision to the machine pool for the zone.
   • If you deployed your cluster using multiple availability zones, select a Compute node count (per zone) from the drop-down menu. This defines the number of compute nodes to provision to the machine pool per zone.
7. Optional: Configure Root disk size.

8. Optional: Add node labels and taints for your machine pool:

   1. Expand the Edit node labels and taints menu.
   2. Under Node labels, add Key and Value entries for your node labels.

   3. Under Taints, add Key and Value entries for your taints.

      Note

      Creating a machine pool with taints is only possible if the cluster already has at least one machine pool without a taint.

   4. For each taint, select an Effect from the drop-down menu. Available options include NoSchedule, PreferNoSchedule, and NoExecute.

      Note

      Alternatively, you can add the node labels and taints after you create the machine pool.

9. Optional: Select additional custom security groups to use for nodes in this machine pool. You must have already created the security groups and associated them with the VPC that you selected for this cluster. You cannot add or edit security groups after you create the machine pool. For more information, see the requirements for security groups in the "Additional resources" section.

   Important

   You can use up to ten additional security groups for machine pools on ROSA with HCP clusters.

10. Optional: Use Amazon EC2 Spot Instances if you want to configure your machine pool to deploy machines as non-guaranteed AWS Spot Instances:

    1. Select Use Amazon EC2 Spot Instances.

    2. Leave Use On-Demand instance price selected to use the on-demand instance price. Alternatively, select Set maximum price to define a maximum hourly price for a Spot Instance.

       For more information about Amazon EC2 Spot Instances, see the AWS documentation.

       Important

       Your Amazon EC2 Spot Instances might be interrupted at any time. Use Amazon EC2 Spot Instances only for workloads that can tolerate interruptions.

       Note

       If you select Use Amazon EC2 Spot Instances for a machine pool, you cannot disable the option after the machine pool is created.

11. Click Add machine pool to create the machine pool.

1. List the available machine pools on your cluster:

   $ rosa list machinepools --cluster=<cluster_name>

   Example output

   ID             AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS                  TAINTS    AVAILABILITY ZONES                    SPOT INSTANCES
   Default        No           3         m5.xlarge                                        us-east-1a, us-east-1b, us-east-1c    N/A
   mymachinepool  Yes          3-6       m5.xlarge      app=db, tier=backend              us-east-1a, us-east-1b, us-east-1c    No

2. Describe the information of a specific machine pool in your cluster:

   $ rosa describe machinepool --cluster=<cluster_name> mymachinepool

   Example output

   ID:                         mymachinepool
   Cluster ID:                 27iimopsg1mge0m81l0sqivkne2qu6dr
   Autoscaling:                Yes
   Replicas:                   3-6
   Instance type:              m5.xlarge
   Labels:                     app=db, tier=backend
   Taints:
   Availability zones:         us-east-1a, us-east-1b, us-east-1c
   Subnets:
   Spot instances:             No
   Disk size:                  300 GiB
   Security Group IDs:

3. Verify that the machine pool is included in the output and the configuration is as expected.

Procedure for cluster creation

1. From ROSA cluster wizard, navigate to Cluster settings.
2. Navigate to Machine pool step.
3. Select the desired Root disk size.
4. Select Next to continue creating your cluster.

Procedure for machine pool creation

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Navigate to Machine pool tab.
3. Click Add machine pool.
4. Select the desired Root disk size.
5. Select Add machine pool to create the machine pool.

Procedure for machine pool creation

1. Scale up the cluster by executing the following command:

   $ rosa create machinepool --cluster=<cluster_id>  1
                             --disk-size=<disk_size>  2

   1

   Specifies the ID or name of your existing OpenShift cluster

   2

   Specifies the worker node disk size. The value can be in GB, GiB, TB, or TiB. Replace '<disk_size>' with a numeric value and unit, for example '--disk-size=200GiB'. You cannot separate the digit and the unit. No spaces are allowed.

2. Confirm new machine pool disk volume size by logging into the AWS console and find the EC2 virtual machine root volume size.

Procedure

1. From OpenShift Cluster Manager, navigate to the Clusters page and select the cluster that contains the machine pool that you want to delete.
2. On the selected cluster, select the Machine pools tab.
3. Under the Machine pools tab, click the options menu for the machine pool that you want to delete.
4. Click Delete.

Procedure

1. From the ROSA CLI, run the following command:

   $ rosa delete machinepool -c=<cluster_name> <machine_pool_ID>

   Example output

   ? Are you sure you want to delete machine pool <machine_pool_ID> on cluster <cluster_name>? (y/N)

2. Enter 'y' to delete the machine pool.

   The selected machine pool is deleted.

Procedure

1. List the machine pools in the cluster:

   $ rosa list machinepools --cluster=<cluster_name>

   Example output

   ID           AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS    TAINTS    AVAILABILITY ZONES    SPOT INSTANCES
   Default      No           2         m5.xlarge                          us-east-1a            N/A
   db-nodes-mp  No           2         m5.xlarge                          us-east-1a            No

2. Add or update the node labels for a machine pool:

   • To add or update node labels for a machine pool that does not use autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --replicas=<replica_count> \ 1
                             --labels=<key>=<value>,<key>=<value> \ 2
                             <machine_pool_id>

     1

     For machine pools that do not use autoscaling, you must provide a replica count when adding node labels. If you do not specify the --replicas argument, you are prompted for a replica count before the command completes. If you deployed Red Hat OpenShift Service on AWS (ROSA) using a single availability zone, the replica count defines the number of compute nodes to provision to the machine pool for the zone. If you deployed your cluster using multiple availability zones, the count defines the total number of compute nodes in the machine pool across all zones and must be a multiple of 3.

     2

     Replace <key>=<value>,<key>=<value> with a comma-delimited list of key-value pairs, for example --labels=key1=value1,key2=value2. This list overwrites any modifications made to node labels on an ongoing basis.

     The following example adds labels to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --replicas=2 --labels=app=db,tier=backend db-nodes-mp

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

   • To add or update node labels for a machine pool that uses autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --min-replicas=<minimum_replica_count> \ 1
                             --max-replicas=<maximum_replica_count> \ 2
                             --labels=<key>=<value>,<key>=<value> \ 3
                             <machine_pool_id>

     1 2

     For machine pools that use autoscaling, you must provide minimum and maximum compute node replica limits. If you do not specify the arguments, you are prompted for the values before the command completes. The cluster autoscaler does not reduce or increase the machine pool node count beyond the limits that you specify. If you deployed ROSA using a single availability zone, the --min-replicas and --max-replicas arguments define the autoscaling limits in the machine pool for the zone. If you deployed your cluster using multiple availability zones, the arguments define the autoscaling limits in total across all zones and the counts must be multiples of 3.

     3

     Replace <key>=<value>,<key>=<value> with a comma-delimited list of key-value pairs, for example --labels=key1=value1,key2=value2. This list overwrites any modifications made to node labels on an ongoing basis.

     The following example adds labels to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --min-replicas=2 --max-replicas=3 --labels=app=db,tier=backend db-nodes-mp

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

Verification

1. Describe the details of the machine pool with the new labels:

   $ rosa describe machinepool --cluster=<cluster_name> <machine-pool-name>

   Example output

   ID:                         db-nodes-mp
   Cluster ID:                 <ID_of_cluster>
   Autoscaling:                No
   Replicas:                   2
   Instance type:              m5.xlarge
   Labels:                     app=db, tier=backend
   Taints:
   Availability zones:         us-east-1a
   Subnets:
   Spot instances:             No
   Disk size:                  300 GiB
   Security Group IDs:

2. Verify that the labels are included for your machine pool in the output.

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Machine pools tab, click the options menu for the machine pool that you want to add a taint to.
3. Select Edit taints.
4. Add Key and Value entries for your taint.
5. Select an Effect for your taint from the drop-down menu. Available options include NoSchedule, PreferNoSchedule, and NoExecute.
6. Optional: Select Add taint if you want to add more taints to the machine pool.
7. Click Save to apply the taints to the machine pool.

Verification

1. Under the Machine pools tab, select > next to your machine pool to expand the view.
2. Verify that your taints are listed under Taints in the expanded view.

Procedure

1. List the machine pools in the cluster by running the following command:

   $ rosa list machinepools --cluster=<cluster_name>

   Example output

   ID           AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS    TAINTS    AVAILABILITY ZONES    SPOT INSTANCES     DISK SIZE   SG IDs
   Default      No           2         m5.xlarge                          us-east-1a            N/A                300 GiB     sg-0e375ff0ec4a6cfa2
   db-nodes-mp  No           2         m5.xlarge                          us-east-1a            No                 300 GiB     sg-0e375ff0ec4a6cfa2

2. Add or update the taints for a machine pool:

   • To add or update taints for a machine pool that does not use autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --replicas=<replica_count> \ 1
                             --taints=<key>=<value>:<effect>,<key>=<value>:<effect> \ 2
                             <machine_pool_id>

     1

     For machine pools that do not use autoscaling, you must provide a replica count when adding taints. If you do not specify the --replicas argument, you are prompted for a replica count before the command completes. If you deployed Red Hat OpenShift Service on AWS (ROSA) using a single availability zone, the replica count defines the number of compute nodes to provision to the machine pool for the zone. If you deployed your cluster using multiple availability zones, the count defines the total number of compute nodes in the machine pool across all zones and must be a multiple of 3.

     2

     Replace <key>=<value>:<effect>,<key>=<value>:<effect> with a key, value, and effect for each taint, for example --taints=key1=value1:NoSchedule,key2=value2:NoExecute. Available effects include NoSchedule, PreferNoSchedule, and NoExecute.This list overwrites any modifications made to node taints on an ongoing basis.

     The following example adds taints to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --replicas 2 --taints=key1=value1:NoSchedule,key2=value2:NoExecute db-nodes-mp

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

   • To add or update taints for a machine pool that uses autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --min-replicas=<minimum_replica_count> \ 1
                             --max-replicas=<maximum_replica_count> \ 2
                             --taints=<key>=<value>:<effect>,<key>=<value>:<effect> \ 3
                             <machine_pool_id>

     1 2

     For machine pools that use autoscaling, you must provide minimum and maximum compute node replica limits. If you do not specify the arguments, you are prompted for the values before the command completes. The cluster autoscaler does not reduce or increase the machine pool node count beyond the limits that you specify. If you deployed ROSA using a single availability zone, the --min-replicas and --max-replicas arguments define the autoscaling limits in the machine pool for the zone. If you deployed your cluster using multiple availability zones, the arguments define the autoscaling limits in total across all zones and the counts must be multiples of 3.

     3

     Replace <key>=<value>:<effect>,<key>=<value>:<effect> with a key, value, and effect for each taint, for example --taints=key1=value1:NoSchedule,key2=value2:NoExecute. Available effects include NoSchedule, PreferNoSchedule, and NoExecute. This list overwrites any modifications made to node taints on an ongoing basis.

     The following example adds taints to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --min-replicas=2 --max-replicas=3 --taints=key1=value1:NoSchedule,key2=value2:NoExecute db-nodes-mp

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

Verification

1. Describe the details of the machine pool with the new taints:

   $ rosa describe machinepool --cluster=<cluster_name> <machine-pool-name>

   Example output

   ID:                         db-nodes-mp
   Cluster ID:                 <ID_of_cluster>
   Autoscaling:                No
   Replicas:                   2
   Instance type:              m5.xlarge
   Labels:
   Taints:                     key1=value1:NoSchedule, key2=value2:NoExecute
   Availability zones:         us-east-1a
   Subnets:
   Spot instances:             No
   Disk size:                  300 GiB
   Security Group IDs:

2. Verify that the taints are included for your machine pool in the output.

1. Determine expected container memory usage

   Determine expected mean and peak container memory usage, empirically if necessary (for example, by separate load testing). Remember to consider all the processes that may potentially run in parallel in the container: for example, does the main application spawn any ancillary scripts?

2. Determine risk appetite

   Determine risk appetite for eviction. If the risk appetite is low, the container should request memory according to the expected peak usage plus a percentage safety margin. If the risk appetite is higher, it may be more appropriate to request memory according to the expected mean usage.

3. Set container memory request

   Set container memory request based on the above. The more accurately the request represents the application memory usage, the better. If the request is too high, cluster and quota usage will be inefficient. If the request is too low, the chances of application eviction increase.

4. Set container memory limit, if required

   Set container memory limit, if required. Setting a limit has the effect of immediately killing a container process if the combined memory usage of all processes in the container exceeds the limit, and is therefore a mixed blessing. On the one hand, it may make unanticipated excess memory usage obvious early ("fail fast"); on the other hand it also terminates processes abruptly.

   Note that some Red Hat OpenShift Service on AWS clusters may require a limit value to be set; some may override the request based on the limit; and some application images rely on a limit value being set as this is easier to detect than a request value.

   If the memory limit is set, it should not be set to less than the expected peak container memory usage plus a percentage safety margin.

5. Ensure application is tuned

   Ensure application is tuned with respect to configured request and limit values, if appropriate. This step is particularly relevant to applications which pool memory, such as the JVM. The rest of this page discusses this.