Cluster administration

Procedure

1. Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard, and navigate to the VPC Dashboard.
2. Under Virtual private cloud click on Your VPCs and identify the name and VPC ID for the VPC containing the Red Hat OpenShift Service on AWS cluster.
3. Under Virtual private network (VPN) click Customer gateways.
4. Click Create customer gateway and give it a meaningful name.
5. Enter the ASN of your customer gateway device in the BGP ASN field.
6. Enter the IP address for your customer gateway devices’s external interface in the IP address field.
7. Click Create customer gateway.

8. If you do not already have a Virtual Private Gateway attached to the intended VPC:

   1. From the VPC Dashboard, click on Virtual Private Gateways.
   2. Click Create virtual private gateway, give it a meaningful name.
   3. Click Create virtual private gateway, leaving the Amazon default ASN.
   4. Select the newly created gateway.
   5. Select Actions from the list and click Attach to VPC.
   6. Select the newly created gateway under Available VPC’s and click Attach to VPC to attach it to the cluster VPC you identified earlier.

Procedure

1. From the VPC dashboard, under Virtual private network (VPN) click on Site-to-Site VPN connections.

2. Click Create VPN connection.

   1. Give it a meaningful name tag.
   2. Select the Virtual private gateway created previously.
   3. For Customer gateway, select Existing.
   4. Select the Customer gateway id by name.
   5. If the VPN will use BGP, select Dynamic, otherwise select Static and enter the Static IP CIDRs. If there are multiple CIDRs, add each CIDR as Another Rule.
   6. Click Create VPN connection.
   7. Under State wait for the VPN status to change from Pending to Available, approximately 5 to 10 minutes.

3. Select the VPN you just created and click Download configuration.

   1. From the list, select the vendor, platform, and version of the customer gateway device, then click Download.
   2. The Generic vendor configuration is also available for retrieving information in a plain text format.

Procedure

1. From the VPC Dashboard, under Virtual private cloud, click on Route tables.

2. Select the private Route table associated with the VPC that contains your Red Hat OpenShift Service on AWS cluster.

   Note

   On some clusters, there may be more than one route table for a particular VPC. Select the private one that has a number of explicitly associated subnets.

3. Click on the Route Propagation tab.

4. In the table that appears, you should see the Virtual Private Gateway you created previously. Check the value in the Propagate column.

   1. If Propagation is set to No, click Edit route propagation, check the Enable checkbox in Propagation and click Save.

Procedure

1. Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region.
2. Select Direct Connect from the Services menu.
3. There will be one or more Virtual Interfaces waiting to be accepted, select one of them to view the Summary.
4. View the Virtual Interface type: private or public.
5. Record the Amazon side ASN value.

Procedure

1. Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region.
2. From the AWS region, select VPC from the Services menu.
3. From Virtual private network (VPN), select Virtual private gateways.
4. Click Create virtual private gateway.
5. Give the Virtual Private Gateway a suitable name.
6. Select Custom ASN in the Enter custom ASN field enter the Amazon side ASN value gathered previously.
7. Click Create virtual private gateway.
8. Click the newly created Virtual Private Gateway and choose Attach to VPC from the Actions tab.
9. Select the Red Hat OpenShift Service on AWS Cluster VPC from the list, and click Attach VPC.

Procedure

1. Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region.
2. From the Red Hat OpenShift Service on AWS AWS Account region, select Direct Connect from the Services menu.
3. Select Direct Connect gateways and Create Direct Connect gateway.
4. Give the Direct Connect gateway a suitable name.
5. In the Amazon side ASN, enter the Amazon side ASN value gathered previously.
6. Click Create the Direct Connect gateway.

Procedure

1. Log in to the Red Hat OpenShift Service on AWS AWS Account Dashboard and select the correct region.
2. From the Red Hat OpenShift Service on AWS AWS Account region, select Direct Connect from the Services menu.
3. Select one of the Direct Connect Virtual Interfaces from the list.
4. Check the Interface State has become Available
5. Check the Interface BGP Status has become Up.
6. Repeat this verification for any remaining Direct Connect Interfaces.

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Machine pools tab, click Add machine pool.
3. Add a Machine pool name.

4. Select a Compute node instance type from the list. The instance type defines the vCPU and memory allocation for each compute node in the machine pool.

   Note

   You cannot change the instance type for a machine pool after the pool is created.

5. Optional: Configure autoscaling for the machine pool:

   1. Select Enable autoscaling to automatically scale the number of machines in your machine pool to meet the deployment needs.

   2. Set the minimum and maximum node count limits for autoscaling. The cluster autoscaler does not reduce or increase the machine pool node count beyond the limits that you specify.

      Note

      Alternatively, you can set your autoscaling preferences for the machine pool after the machine pool is created.

6. If you did not enable autoscaling, select a Compute node count from the drop-down menu. This defines the number of compute nodes to provision to the machine pool for the availability zone.
7. Optional: Configure Root disk size.

8. Optional: Add node labels and taints for your machine pool:

   1. Expand the Edit node labels and taints menu.
   2. Under Node labels, add Key and Value entries for your node labels.

   3. Under Taints, add Key and Value entries for your taints.

      Note

      Creating a machine pool with taints is only possible if the cluster already has at least one machine pool without a taint.

   4. For each taint, select an Effect from the drop-down menu. Available options include NoSchedule, PreferNoSchedule, and NoExecute.

      Note

      Alternatively, you can add the node labels and taints after you create the machine pool.

9. Optional: Select additional custom security groups to use for nodes in this machine pool. You must have already created the security groups and associated them with the VPC that you selected for this cluster. You cannot add or edit security groups after you create the machine pool.

   Important

   You can use up to ten additional security groups for machine pools on Red Hat OpenShift Service on AWS clusters.

10. Click Add machine pool to create the machine pool.

1. List the available machine pools on your cluster:

   $ rosa list machinepools --cluster=<cluster_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID             AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS                  TAINTS    AVAILABILITY ZONE  SUBNET                    VERSION  AUTOREPAIR
   Default        No           1/1       m5. xlarge                                       us-east-2c         subnet-00552ad67728a6ba3  4.14.34  Yes
   mymachinepool  Yes          3/3-6     m5.xlarge      app=db, tier=backend              us-east-2a         subnet-0cb56f5f41880c413  4.14.34  Yes

   Copy to Clipboard Toggle word wrap

2. Describe the information of a specific machine pool in your cluster:

   $ rosa describe machinepool --cluster=<cluster_name> --machinepool=mymachinepool

   Copy to Clipboard Toggle word wrap

   Example output

   ID:                         mymachinepool
   Cluster ID:                 2d6010rjvg17anri30v84vspf7c7kr6v
   Autoscaling:                Yes
   Desired replicas:           3-6
   Current replicas:           3
   Instance type:              m5.xlarge
   Labels:                     app=db, tier=backend
   Taints:
   Availability zone:          us-east-2a
   Subnet:                     subnet-0cb56f5f41880c413
   Version:                    4.14.34
   Autorepair:                 Yes
   Tuning configs:
   Additional security group IDs:
   Node drain grace period:
   Message:

   Copy to Clipboard Toggle word wrap

3. Verify that the machine pool is included in the output and the configuration is as expected.

Procedure for machine pool creation

1. Scale up the cluster by executing the following command:

   $ rosa create machinepool --cluster=<cluster_id> \

   1

   
                             --disk-size=<disk_size> 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specifies the ID or name of your existing OpenShift cluster.

   2

   Specifies the worker node disk size. The value can be in GB, GiB, TB, or TiB. Replace <disk_size> with a numeric value and unit, for example --disk-size=200GiB. You cannot separate the digit and the unit. No spaces are allowed.

2. Confirm new machine pool disk volume size by logging into the AWS console and find the EC2 virtual machine root volume size.

Procedure

1. From OpenShift Cluster Manager, navigate to the Cluster List page and select the cluster that contains the machine pool that you want to delete.
2. On the selected cluster, select the Machine pools tab.
3. Under the Machine pools tab, click the Options menu for the machine pool that you want to delete.

4. Click Delete.

   The selected machine pool is deleted.

Procedure

1. From the ROSA CLI, run the following command:

   $ rosa delete machinepool -c=<cluster_name> <machine_pool_ID>

   Copy to Clipboard Toggle word wrap

   Example output

   ? Are you sure you want to delete machine pool <machine_pool_ID> on cluster <cluster_name>? (y/N)

   Copy to Clipboard Toggle word wrap

2. Enter y to delete the machine pool.

   The selected machine pool is deleted.

Procedure

1. List the machine pools in the cluster:

   $ rosa list machinepools --cluster=<cluster_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID           AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS    TAINTS    AVAILABILITY ZONE  SUBNET                    VERSION  AUTOREPAIR
   workers      No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes
   db-nodes-mp  No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes

   Copy to Clipboard Toggle word wrap

2. Add or update the node labels for a machine pool:

   • To add or update node labels for a machine pool that does not use autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --labels=<key>=<value>,<key>=<value> \

     1

     
                             <machine_pool_id>

     Copy to Clipboard Toggle word wrap

     1

     Replace <key>=<value>,<key>=<value> with a comma-delimited list of key-value pairs, for example --labels=key1=value1,key2=value2. This list overwrites any modifications made to node labels on an ongoing basis.

     The following example adds labels to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --replicas=2 --labels=app=db,tier=backend db-nodes-mp

     Copy to Clipboard Toggle word wrap

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

     Copy to Clipboard Toggle word wrap

Verification

1. Describe the details of the machine pool with the new labels:

   $ rosa describe machinepool --cluster=<cluster_name> --machinepool=<machine-pool-name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID:                            db-nodes-mp
   Cluster ID:                    <ID_of_cluster>
   Autoscaling:                   No
   Desired replicas:              2
   Current replicas:              2
   Instance type:                 m5.xlarge
   Labels:                        app=db, tier=backend
   Tags:
   Taints:
   Availability zone:             us-east-2a
   Subnet:                        subnet-0df2ec3377847164f
   Disk size:                     300 GiB
   Version:                       4.16.6
   EC2 Metadata Http Tokens:      optional
   Autorepair:                    Yes
   Tuning configs:
   Kubelet configs:
   Additional security group IDs:
   Node drain grace period:
   Management upgrade:
    - Type:                       Replace
    - Max surge:                  1
    - Max unavailable:            0
   Message:

   Copy to Clipboard Toggle word wrap

2. Verify that the labels are included for your machine pool in the output.

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Machine pools tab, click the Options menu for the machine pool that you want to add a taint to.
3. Select Edit taints.
4. Add Key and Value entries for your taint.
5. Select an Effect for your taint from the list. Available options include NoSchedule, PreferNoSchedule, and NoExecute.
6. Optional: Select Add taint if you want to add more taints to the machine pool.
7. Click Save to apply the taints to the machine pool.

Verification

1. Under the Machine pools tab, select > next to your machine pool to expand the view.
2. Verify that your taints are listed under Taints in the expanded view.

Procedure

1. List the machine pools in the cluster by running the following command:

   $ rosa list machinepools --cluster=<cluster_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID           AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS    TAINTS    AVAILABILITY ZONE  SUBNET                    VERSION  AUTOREPAIR
   workers      No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes
   db-nodes-mp  No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes

   Copy to Clipboard Toggle word wrap

2. Add or update the taints for a machine pool:

   • To add or update taints for a machine pool that does not use autoscaling, run the following command:

     $ rosa edit machinepool --cluster=<cluster_name> \
                             --taints=<key>=<value>:<effect>,<key>=<value>:<effect> \

     1

     
                             <machine_pool_id>

     Copy to Clipboard Toggle word wrap

     1

     Replace <key>=<value>:<effect>,<key>=<value>:<effect> with a key, value, and effect for each taint, for example --taints=key1=value1:NoSchedule,key2=value2:NoExecute. Available effects include NoSchedule, PreferNoSchedule, and NoExecute.This list overwrites any modifications made to node taints on an ongoing basis.

     The following example adds taints to the db-nodes-mp machine pool:

     $ rosa edit machinepool --cluster=mycluster --replicas 2 --taints=key1=value1:NoSchedule,key2=value2:NoExecute db-nodes-mp

     Copy to Clipboard Toggle word wrap

     Example output

     I: Updated machine pool 'db-nodes-mp' on cluster 'mycluster'

     Copy to Clipboard Toggle word wrap

Verification

1. Describe the details of the machine pool with the new taints:

   $ rosa describe machinepool --cluster=<cluster_name> --machinepool=<machinepool_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID:                            db-nodes-mp
   Cluster ID:                    <ID_of_cluster>
   Autoscaling:                   No
   Desired replicas:              2
   Current replicas:              2
   Instance type:                 m5.xlarge
   Labels:
   Tags:
   Taints:                        key1=value1:NoSchedule, key2=value2:NoExecute
   Availability zone:             us-east-2a
   ...

   Copy to Clipboard Toggle word wrap

2. Verify that the taints are included for your machine pool in the output.

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Machine pools tab, click the Options menu for the machine pool that you want to configure auto repair for.
3. From the menu, select Edit.
4. From the Edit Machine Pool dialog box that displays, find the AutoRepair option.
5. Select or clear the box next to AutoRepair to enable or disable.
6. Click Save to apply the change to the machine pool.

Verification

1. Under the Machine pools tab, select > next to your machine pool to expand the view.
2. Verify that your machine pool has the correct AutoRepair setting in the expanded view.

Procedure

1. List the machine pools in the cluster by running the following command:

   $ rosa list machinepools --cluster=<cluster_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID           AUTOSCALING  REPLICAS  INSTANCE TYPE  LABELS    TAINTS    AVAILABILITY ZONE  SUBNET                    VERSION  AUTOREPAIR
   workers      No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes
   db-nodes-mp  No           2/2       m5.xlarge                          us-east-2a         subnet-0df2ec3377847164f  4.16.6   Yes

   Copy to Clipboard Toggle word wrap

2. Enable or disable AutoRepair on a machine pool:

   • To disable AutoRepair for a machine pool, run the following command:

     $ rosa edit machinepool --cluster=mycluster --machinepool=<machinepool_name>  --autorepair=false

     Copy to Clipboard Toggle word wrap

   • To enable AutoRepair for a machine pool, run the following command:

     $ rosa edit machinepool --cluster=mycluster --machinepool=<machinepool_name>  --autorepair=true

     Copy to Clipboard Toggle word wrap

     Example output

     I: Updated machine pool 'machinepool_name' on cluster 'mycluster'

     Copy to Clipboard Toggle word wrap

Verification

1. Describe the details of the machine pool:

   $ rosa describe machinepool --cluster=<cluster_name> --machinepool=<machinepool_name>

   Copy to Clipboard Toggle word wrap

   Example output

   ID:                            machinepool_name
   Cluster ID:                    <ID_of_cluster>
   Autoscaling:                   No
   Desired replicas:              2
   Current replicas:              2
   Instance type:                 m5.xlarge
   Labels:
   Tags:
   Taints:
   Availability zone:             us-east-2a
   ...
   Autorepair:                    Yes
   Tuning configs:
   Kubelet configs:
   Additional security group IDs:
   Node drain grace period:
   Management upgrade:
    - Type:                               Replace
    - Max surge:                          1
    - Max unavailable:                    0

   Copy to Clipboard Toggle word wrap

2. Verify that the AutoRepair setting is correct for your machine pool in the output.

1. Determine expected container memory usage

   Determine expected mean and peak container memory usage, empirically if necessary (for example, by separate load testing). Remember to consider all the processes that may potentially run in parallel in the container: for example, does the main application spawn any ancillary scripts?

2. Determine risk appetite

   Determine risk appetite for eviction. If the risk appetite is low, the container should request memory according to the expected peak usage plus a percentage safety margin. If the risk appetite is higher, it may be more appropriate to request memory according to the expected mean usage.

3. Set container memory request

   Set container memory request based on the above. The more accurately the request represents the application memory usage, the better. If the request is too high, cluster and quota usage will be inefficient. If the request is too low, the chances of application eviction increase.

4. Set container memory limit, if required

   Set container memory limit, if required. Setting a limit has the effect of immediately killing a container process if the combined memory usage of all processes in the container exceeds the limit, and is therefore a mixed blessing. On the one hand, it may make unanticipated excess memory usage obvious early ("fail fast"); on the other hand it also terminates processes abruptly.

   Note that some Red Hat OpenShift Service on AWS clusters may require a limit value to be set; some may override the request based on the limit; and some application images rely on a limit value being set as this is easier to detect than a request value.

   If the memory limit is set, it should not be set to less than the expected peak container memory usage plus a percentage safety margin.

5. Ensure application is tuned

   Ensure application is tuned with respect to configured request and limit values, if appropriate. This step is particularly relevant to applications which pool memory, such as the JVM. The rest of this page discusses this.