Chapter 16. Extending an AWS VPC cluster into an AWS Outpost

Procedure

1. List the infrastructure ID for the cluster by running the following command. Retain this value.

   $ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructures.config.openshift.io cluster

2. Obtain details about the compute machine sets that the installation program created by running the following commands:

   1. List the compute machine sets on your cluster:

      $ oc get machinesets.machine.openshift.io -n openshift-machine-api

      Example output

      NAME                           DESIRED   CURRENT   READY   AVAILABLE   AGE
      <compute_machine_set_name_1>   1         1         1       1           55m
      <compute_machine_set_name_2>   1         1         1       1           55m

   2. Display the Amazon Machine Image (AMI) ID for one of the listed compute machine sets. Retain this value.

      $ oc get machinesets.machine.openshift.io <compute_machine_set_name_1> \
        -n openshift-machine-api \
        -o jsonpath='{.spec.template.spec.providerSpec.value.ami.id}'

   3. Display the subnet ID for the AWS VPC cluster. Retain this value.

      $ oc get machinesets.machine.openshift.io <compute_machine_set_name_1> \
        -n openshift-machine-api \
        -o jsonpath='{.spec.template.spec.providerSpec.value.subnet.id}'

Procedure

1. List the Outposts that are connected to your AWS account by running the following command:

   $ aws outposts list-outposts

2. Retain the following values from the output of the aws outposts list-outposts command:

   • The Outpost ID.
   • The Amazon Resource Name (ARN) for the Outpost.

   • The Outpost availability zone.

     Note

     The output of the aws outposts list-outposts command includes two values related to the availability zone: AvailabilityZone and AvailabilityZoneId. You use the AvailablilityZone value to configure a compute machine set that creates compute machines in your Outpost.

3. Using the value of the Outpost ID, show the instance types that are available in your Outpost by running the following command. Retain the values of the available instance types.

   $ aws outposts get-outpost-instance-types \
     --outpost-id <outpost_id_value>

4. Using the value of the Outpost ARN, show the subnet ID for the Outpost by running the following command. Retain this value.

   $ aws ec2 describe-subnets \
     --filters Name=outpost-arn,Values=<outpost_arn_value>

Procedure

1. To obtain the current MTU for the cluster network, enter the following command:

   $ oc describe network.config cluster

   Example output

   ...
   Status:
     Cluster Network:
       Cidr:               10.217.0.0/22
       Host Prefix:        23
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       10.217.4.0/23
   ...

2. To begin the MTU migration, specify the migration configuration by entering the following command. The Machine Config Operator performs a rolling reboot of the nodes in the cluster in preparation for the MTU change.

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }'

   where:

   <overlay_from>

   Specifies the current cluster network MTU value.

   <overlay_to>

   Specifies the target MTU for the cluster network. This value is set relative to the value of <machine_to>. For OVN-Kubernetes, this value must be 100 less than the value of <machine_to>.

   <machine_to>

   Specifies the MTU for the primary network interface on the underlying host network.

   Example that decreases the cluster MTU

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 1000 } , "machine": { "to" : 1100} } } } }'

3. As the Machine Config Operator updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get machineconfigpools

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the Machine Config Operator updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

4. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

   2. Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   3. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/mtu-migration.sh

5. To finalize the MTU migration, enter the following command for the OVN-Kubernetes network plugin:

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}'

   where:

   <mtu>

   Specifies the new cluster network MTU that you specified with <overlay_to>.

6. After finalizing the MTU migration, each machine config pool node is rebooted one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get machineconfigpools

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

Procedure

1. Go to the section of the documentation named "CloudFormation template for the VPC subnet", and copy the syntax from the template. Save the copied template syntax as a YAML file on your local system. This template describes the VPC that your cluster requires.

2. Run the following command to deploy the CloudFormation template, which creates a stack of AWS resources that represent the VPC:

   $ aws cloudformation create-stack --stack-name <stack_name> \1
     --region ${CLUSTER_REGION} \
     --template-body file://<template>.yaml \2
     --parameters \
       ParameterKey=VpcId,ParameterValue="${VPC_ID}" \3
       ParameterKey=ClusterName,ParameterValue="${CLUSTER_NAME}" \4
       ParameterKey=ZoneName,ParameterValue="${ZONE_NAME}" \5
       ParameterKey=PublicRouteTableId,ParameterValue="${ROUTE_TABLE_PUB}" \6
       ParameterKey=PublicSubnetCidr,ParameterValue="${SUBNET_CIDR_PUB}" \7
       ParameterKey=PrivateRouteTableId,ParameterValue="${ROUTE_TABLE_PVT}" \8
       ParameterKey=PrivateSubnetCidr,ParameterValue="${SUBNET_CIDR_PVT}" \9
       ParameterKey=PrivateSubnetLabel,ParameterValue="private-outpost" \
       ParameterKey=PublicSubnetLabel,ParameterValue="public-outpost" \
       ParameterKey=OutpostArn,ParameterValue="${OUTPOST_ARN}" 10

   1

   <stack_name> is the name for the CloudFormation stack, such as cluster-<outpost_name>.

   2

   <template> is the relative path and the name of the CloudFormation template YAML file that you saved.

   3

   ${VPC_ID} is the VPC ID, which is the value VpcID in the output of the CloudFormation template for the VPC.

   4

   ${CLUSTER_NAME} is the value of ClusterName to be used as a prefix of the new AWS resource names.

   5

   ${ZONE_NAME} is the value of AWS Outposts name to create the subnets.

   6

   ${ROUTE_TABLE_PUB} is the Public Route Table ID created in the ${VPC_ID} used to associate the public subnets on Outposts. Specify the public route table to associate the Outpost subnet created by this stack.

   7

   ${SUBNET_CIDR_PUB} is a valid CIDR block that is used to create the public subnet. This block must be part of the VPC CIDR block VpcCidr.

   8

   ${ROUTE_TABLE_PVT} is the Private Route Table ID created in the ${VPC_ID} used to associate the private subnets on Outposts. Specify the private route table to associate the Outpost subnet created by this stack.

   9

   ${SUBNET_CIDR_PVT} is a valid CIDR block that is used to create the private subnet. This block must be part of the VPC CIDR block VpcCidr.

   10

   ${OUTPOST_ARN} is the Amazon Resource Name (ARN) for the Outpost.

   Example output

   arn:aws:cloudformation:us-east-1:123456789012:stack/<stack_name>/dbedae40-820e-11eb-2fd3-12a48460849f

Procedure

1. Optional: Verify that the edge compute nodes have the location=outposts label by running the following command and verifying that the output includes only the edge compute nodes in your Outpost:

   $ oc get nodes -l location=outposts

2. Label the cloud-based compute nodes in the VPC cluster with a key-value pair by running the following command:

   $ for NODE in $(oc get node -l node-role.kubernetes.io/worker --no-headers | grep -v outposts | awk '{print$1}'); do oc label node $NODE <key_name>=<value>; done

   where <key_name>=<value> is the label you want to use to distinguish cloud-based compute nodes.

   Example output

   node1.example.com labeled
   node2.example.com labeled
   node3.example.com labeled

3. Optional: Verify that the cloud-based compute nodes have the specified label by running the following command and confirming that the output includes all cloud-based compute nodes in your VPC cluster:

   $ oc get nodes -l <key_name>=<value>

   Example output

   NAME                   STATUS    ROLES     AGE       VERSION
   node1.example.com      Ready     worker    7h        v1.28.5
   node2.example.com      Ready     worker    7h        v1.28.5
   node3.example.com      Ready     worker    7h        v1.28.5

4. Configure the Classic Load Balancer service by adding the cloud-based subnet information to the annotations field of the Service manifest:

   Example service configuration

   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: <application_name>
     name: <application_name>
     namespace: <application_namespace>
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-subnets: <aws_subnet> 1
       service.beta.kubernetes.io/aws-load-balancer-target-node-labels: <key_name>=<value> 2
   spec:
     ports:
     - name: http
       port: 80
       protocol: TCP
       targetPort: 8080
     selector:
       app: <application_name>
     type: LoadBalancer

   1

   Specify the subnet ID for the AWS VPC cluster.

   2

   Specify the key-value pair that matches the pair in the node label.

5. Create the Service CR by running the following command:

   $ oc create -f <file_name>.yaml

Verification

1. Verify the status of the service resource to show the host of the provisioned Classic Load Balancer by running the following command:

   $ HOST=$(oc get service <application_name> -n <application_namespace> --template='{{(index .status.loadBalancer.ingress 0).hostname}}')

2. Verify the status of the provisioned Classic Load Balancer host by running the following command:

   $ curl $HOST

3. In the AWS console, verify that only the labeled instances appear as the targeted instances for the load balancer.