Chapter 3. Deploying OpenShift sandboxed containers workloads using peer pods

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Retrieve the Instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

3. Retrieve the security groups:

   $ SG=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region ${AWS_REGION})

4. Authorize the peer-pods shim and to access kata-agent communication. Run the following command:

   $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 15150 --source-group ${SG} --region ${AWS_REGION}

5. Set up the peer-pods tunnel. Run the following command:

   $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 9000 --source-group ${SG} --region ${AWS_REGION}

Procedure

1. Retrieve the Instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

2. Retrieve the Azure resource group:

   $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

3. Retrieve the Azure network security group (NSG) name:

   $ AZURE_NSG_NAME=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

4. Retrieve the Azure VNet name:

   $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

5. Retrieve the Azure subnet name:

   $ AZURE_SUBNET_NAME=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name ${AZURE_VNET_NAME}  --query "[].{Name:name} | [? contains(Name, 'worker')]" --output tsv)

6. Retrieve the Azure subnet prefix:

   $ AZURE_SUBNET_PREFIX=$(az network vnet subnet show --name ${AZURE_SUBNET_NAME} --vnet-name ${AZURE_VNET_NAME} --resource-group ${AZURE_RESOURCE_GROUP} --query "addressPrefix" --output tsv)

7. Authorize the peer-pods shim to access kata-agent communication. Run the following command:

   $ az network nsg rule create \
     --resource-group $AZURE_RESOURCE_GROUP \
     --nsg-name $AZURE_NSG_NAME \
     --name Allow-Kata-Agent-Internal \
     --access Allow \
     --protocol Tcp \
     --direction Inbound \
     --priority 112 \
     --source-address-prefixes $AZURE_SUBNET_PREFIX \
     --source-port-range "*" \
     --destination-address-prefixes $AZURE_SUBNET_PREFIX  \
     --destination-port-range 15150

8. Set up the peer-pods tunnel. Run the following command:

   $ az network nsg rule create \
     --resource-group $AZURE_RESOURCE_GROUP \
     --nsg-name $AZURE_NSG_NAME \
     --name Allow-VXLAN-Internal \
     --access Allow \
     --protocol Tcp \
     --direction Inbound \
     --priority 111 \
     --source-address-prefixes $AZURE_SUBNET_PREFIX \
     --source-port-range "*" \
     --destination-address-prefixes $AZURE_SUBNET_PREFIX  \
     --destination-port-range 9000

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<enter value>" 1
     AWS_SECRET_ACCESS_KEY: "<enter value>" 2
     AWS_REGION: "<enter value>" 3
     AWS_SUBNET_ID: "<enter value>" 4
     AWS_VPC_ID: "<enter value>" 5
     AWS_SG_IDS: "<enter value>" 6

   1

   Enter the AWS_ACCESS_KEY_ID value you prepared before you began.

   2

   Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.

   3

   Enter the AWS_REGION value you retrieved.

   4

   Enter the AWS_SUBNET_ID value you retrieved.

   5

   Enter the AWS_VPC_ID value you retrieved.

   6

   Enter the AWS_SG_IDS value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Workloads Jobs.
2. In the Jobs window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
3. Click Create Job.
4. In the Create Job window, paste this complete YAML manifest.
5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium"
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<enter value>" 1

   1

   Enter the PODVM_AMI_ID value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<enter value>" 1
     AZURE_CLIENT_SECRET: "<enter value>" 2
     AZURE_TENANT_ID: "<enter value>" 3
     AZURE_SUBSCRIPTION_ID: "<enter value>" 4
     AZURE_REGION: "<enter value>" 5
     AZURE_RESOURCE_GROUP: "<enter value>" 6

   1

   Enter the AZURE_CLIENT_ID value you prepared before you began.

   2

   Enter the AZURE_CLIENT_SECRET value you prepared before you began.

   3

   Enter the AZURE_TENANT_ID value you prepared before you began.

   4

   Enter the AZURE_SUBSCRIPTION_ID value you retrieved.

   5

   Enter the AZURE_REGION value you retrieved.

   6

   Enter the AZURE_RESOURCE_GROUP value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Workloads Jobs.
2. In the Jobs window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
3. Click Create Job.
4. In the Create Job window, paste this complete YAML manifest.
5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
     AZURE_SUBNET_ID: "<enter value>" 1
     AZURE_NSG_ID: "<enter value>" 2
     AZURE_IMAGE_ID: "<enter value>" 3
     PROXY_TIMEOUT: "5m"
     DISABLECVM: "true"

   1

   Enter the AZURE_SUBNET_ID value you retrieved.

   2

   Enter the AZURE_NSG_ID value you retrieved.

   3

   Enter the AZURE_IMAGE_ID value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Workloads Secrets.
2. In the Secrets window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
3. Click Create and select Key/value secret from the list.
4. In the Secret name field, enter ssh-key-secret.
5. In the Key field, enter id_rsa.pub.
6. In the Value field, paste your public SSH key.
7. Click Create.

Procedure

1. Collect the necessary parameter values for the secret object. Make sure to write down each value.

   1. Retrieve the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This value is not needed for the secret object itself, but is used to retrieve other values for the secret object.

   2. Retrieve the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

   3. Retrieve the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

   4. Retrieve the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

   5. Retrieve the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region ${AWS_REGION} --output text)
      && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<enter value>" 1
     AWS_SECRET_ACCESS_KEY: "<enter value>" 2
     AWS_REGION: "<enter value>" 3
     AWS_SUBNET_ID: "<enter value>" 4
     AWS_VPC_ID: "<enter value>" 5
     AWS_SG_IDS: "<enter value>" 6

   1

   Enter the AWS_ACCESS_KEY_ID value you prepared before you began.

   2

   Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.

   3

   Enter the AWS_REGION value you retrieved.

   4

   Enter the AWS_SUBNET_ID value you retrieved.

   5

   Enter the AWS_VPC_ID value you retrieved.

   6

   Enter the AWS_SG_IDS value you retrieved.

3. Apply the secret object:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Run the following K8s job to create the image:

   $ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/peer-pods-tech-preview/hack/aws-image-job.yaml

   Note

   This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the AWS web console or AWS CLI tool.

2. Wait for the job to complete:

   $ oc wait --for=condition=complete job.batch/aws-image-creation --timeout=7m -n openshift-sandboxed-containers-operator

Procedure

1. Retrieve the AMI ID. Make sure to save the value for later.

   $ PODVM_AMI_ID=$(aws ec2 describe-images --query "Images[*].[ImageId]" --filters "Name=name,Values=peer-pod-ami" --region ${AWS_REGION} --output text) && echo "PODVM_AMI_ID: \"$PODVM_AMI_ID\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium"
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<enter value>" 1

   1

   Enter the AMI ID value you retrieved.

3. Deploy the ConfigMap:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Collect additional parameter values for the secret object. Make sure to write down each value.

   1. Retrieve the subscription ID:

      $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""

   2. Retrieve the resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

   3. Retrieve the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<enter value>" 1
     AZURE_CLIENT_SECRET: "<enter value>" 2
     AZURE_TENANT_ID: "<enter value>" 3
     AZURE_SUBSCRIPTION_ID: "<enter value>" 4
     AZURE_REGION: "<enter value>" 5
     AZURE_RESOURCE_GROUP: "<enter value>" 6

   1

   Enter the AZURE_CLIENT_ID value you prepared before you began.

   2

   Enter the AZURE_CLIENT_SECRET value you prepared before you began.

   3

   Enter the AZURE_TENANT_ID value you prepared before you began.

   4

   Enter the AZURE_SUBSCRIPTION_ID value you retrieved.

   5

   Enter the AZURE_REGION value you retrieved.

   6

   Enter the AZURE_RESOURCE_GROUP value you retrieved.

3. Apply the secret object:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Run the following K8s job to create the image:

   $ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/peer-pods-tech-preview/hack/azure-image-job.yaml

   Note

   This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the Azure web console or Azure CLI tool.

2. Wait for the job to complete:

   $ oc wait --for=condition=complete job.batch/azure-image-creation --timeout=7m -n openshift-sandboxed-containers-operator

Procedure

1. Collect the configuration values for the Azure peer-pod ConfigMap. Make sure to write down each value.

   1. Retrieve the Azure image ID:

      $ AZURE_IMAGE_ID=$(az image list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id: id} | [? contains(Id, 'peer-pod-vmimage')]" --output tsv) && echo "AZURE_IMAGE_ID: \"$AZURE_IMAGE_ID\""

   2. Retrieve the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is not needed for the ConfigMap, but is used to retrieve the Azure subnet ID.

   3. Retrieve the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

   4. Retrieve the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
     AZURE_SUBNET_ID: "<enter value>" 1
     AZURE_NSG_ID: "<enter value>" 2
     AZURE_IMAGE_ID: "<enter value>" 3
     PROXY_TIMEOUT: "5m"
     DISABLECVM: "true"

   1

   Enter the AZURE_SUBNET_ID value you retrieved.

   2

   Enter the AZURE_NSG_ID value you retrieved.

   3

   Enter the AZURE_IMAGE_ID value you retrieved.

3. Deploy the ConfigMap:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Generate the SSH key:

   $ ssh-keygen -f ./id_rsa -N ""

2. Create the SHH secret object:

   $ oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa