Chapter 3. Deploying OpenShift sandboxed containers workloads using peer pods

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Retrieve the Instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

3. Retrieve the security groups:

   $ SG=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region ${AWS_REGION})

4. Authorize the peer-pods shim and to access kata-agent communication. Run the following command:

   $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 15150 --source-group ${SG} --region ${AWS_REGION}

5. Set up the peer-pods tunnel. Run the following command:

   $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 9000 --source-group ${SG} --region ${AWS_REGION}

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<enter value>" 1
     AWS_SECRET_ACCESS_KEY: "<enter value>" 2
     AWS_REGION: "<enter value>" 3
     AWS_SUBNET_ID: "<enter value>" 4
     AWS_VPC_ID: "<enter value>" 5
     AWS_SG_IDS: "<enter value>" 6

   1

   Enter the AWS_ACCESS_KEY_ID value you prepared before you began.

   2

   Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.

   3

   Enter the AWS_REGION value you retrieved.

   4

   Enter the AWS_SUBNET_ID value you retrieved.

   5

   Enter the AWS_VPC_ID value you retrieved.

   6

   Enter the AWS_SG_IDS value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 1
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
     PROXY_TIMEOUT: "5m"

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<enter value>" 1
     AZURE_CLIENT_SECRET: "<enter value>" 2
     AZURE_TENANT_ID: "<enter value>" 3
     AZURE_SUBSCRIPTION_ID: "<enter value>" 4
     AZURE_REGION: "<enter value>" 5
     AZURE_RESOURCE_GROUP: "<enter value>" 6

   1

   Enter the AZURE_CLIENT_ID value you generated in the RBAC file.

   2

   Enter the AZURE_CLIENT_SECRET value you generated in the RBAC file.

   3

   Enter the AZURE_TENANT_ID value you generated in the RBAC file.

   4

   Enter the AZURE_SUBSCRIPTION_ID value you retrieved.

   5

   Enter the AZURE_REGION value you retrieved.

   6

   Enter the AZURE_RESOURCE_GROUP value you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2
     AZURE_SUBNET_ID: "<enter value>" 3
     AZURE_NSG_ID: "<enter value>" 4
     PROXY_TIMEOUT: "5m"
     DISABLECVM: "true"

   1

   Defines the default instance size that is used when an instance is not defined in the workload.

   2

   List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.

   3

   Enter the AZURE_SUBNET_ID value that you retrieved.

   4

   Enter the AZURE_NSG_ID value that you retrieved.

5. Click Create.

Procedure

1. From the Administrator perspective in the web console, navigate to Workloads Secrets.
2. In the Secrets window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
3. Click Create and select Key/value secret from the list.
4. In the Secret name field, enter ssh-key-secret.
5. In the Key field, enter id_rsa.pub.
6. In the Value field, paste your public SSH key.
7. Click Create.

Procedure

1. Collect the necessary parameter values for the secret object. Make sure to write down each value.

   1. Retrieve the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This value is not needed for the secret object itself, but is used to retrieve other values for the secret object.

   2. Retrieve the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

   3. Retrieve the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

   4. Retrieve the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

   5. Retrieve the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<enter value>" 1
     AWS_SECRET_ACCESS_KEY: "<enter value>" 2
     AWS_REGION: "<enter value>" 3
     AWS_SUBNET_ID: "<enter value>" 4
     AWS_VPC_ID: "<enter value>" 5
     AWS_SG_IDS: "<enter value>" 6

   1

   Enter the AWS_ACCESS_KEY_ID value you prepared before you began.

   2

   Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.

   3

   Enter the AWS_REGION value you retrieved.

   4

   Enter the AWS_SUBNET_ID value you retrieved.

   5

   Enter the AWS_VPC_ID value you retrieved.

   6

   Enter the AWS_SG_IDS value you retrieved.

3. Apply the secret object:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 1
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
     PROXY_TIMEOUT: "5m"

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.

2. Apply the ConfigMap:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Retrieve the Azure subscription ID:

   $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""

2. Generate the RBAC content. This generates the client ID, client secret, and the tenant ID:

   $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query "{ client_id: appId, client_secret: password, tenant_id: tenant }

   The following output appears:

   {
     "client_id": `AZURE_CLIENT_ID`,
     "client_secret": `AZURE_CLIENT_SECRET`,
     "tenant_id": `AZURE_TENANT_ID`
   }

3. Save the values in the RBAC output to use in the secret object.

4. Collect the additional parameter values for the secret object. Make sure to write down each value.

   1. Retrieve the resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

   2. Retrieve the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""

5. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<enter value>" 1
     AZURE_CLIENT_SECRET: "<enter value>" 2
     AZURE_TENANT_ID: "<enter value>" 3
     AZURE_SUBSCRIPTION_ID: "<enter value>" 4
     AZURE_REGION: "<enter value>" 5
     AZURE_RESOURCE_GROUP: "<enter value>" 6

   1

   Enter the AZURE_CLIENT_ID value you generated in the RBAC file.

   2

   Enter the AZURE_CLIENT_SECRET value you generated in the RBAC file.

   3

   Enter the AZURE_TENANT_ID value you generated in the RBAC file.

   4

   Enter the AZURE_SUBSCRIPTION_ID value you retrieved.

   5

   Enter the AZURE_REGION value you retrieved.

   6

   Enter the AZURE_RESOURCE_GROUP value you retrieved.

6. Apply the secret object:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Collect the configuration values for the Azure peer-pod ConfigMap. Make sure to write down each value.

   1. Retrieve the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is not needed for the ConfigMap, but is used to retrieve the Azure subnet ID.

   2. Retrieve the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

   3. Retrieve the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

2. Create a YAML file with the following manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2
     AZURE_SUBNET_ID: "<enter value>" 3
     AZURE_NSG_ID: "<enter value>" 4
     PROXY_TIMEOUT: "5m"
     DISABLECVM: "true"

   1

   Defines the default instance size that is used when an instance is not defined in the workload.

   2

   List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.

   3

   Enter the AZURE_SUBNET_ID value that you retrieved.

   4

   Enter the AZURE_NSG_ID value that you retrieved.

3. Apply the ConfigMap:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Generate the SSH key:

   $ ssh-keygen -f ./id_rsa -N ""

2. Create the SSH secret object:

   $ oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa

Procedure

1. Log in to IBM Z KVM host and set the shell variables to be used by libvirt for storage management.

   1. Set the name of the libvirt pool by running the following command:

      $ export LIBVIRT_POOL=<name_of_libvirt_pool_to_create>

   2. Set the name of the libvirt pool by running the following command:

      $ export LIBVIRT_VOL_NAME=<name_of_libvirt_volume_to_create>

   3. Set the path of the default storage pool location, by running the following command:

      $ export LIBVIRT_POOL_DIRECTORY=<name_of_target_directory> 1

      1

      To ensure libvirt has read and write access permissions, use a subdirectory of libvirt’s storage directory. The default is /var/lib/libvirt/images/.

2. Create a libvirt pool by running the following command:

   $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

3. Start the libvirt pool by running the following command:

   $ virsh pool-start $LIBVIRT_POOL

4. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

Procedure

1. Set the RHEL ORG_ID and ACTIVATION_KEY shell variables for your system.

   1. When using a subscribed RHEL system, set the shell environment variables to the files holding the organization ID and the activation key for Red Hat Subscription Management (RHSM) by running the following commands:

      $ export ORG_ID=$(cat ~/.rh_subscription/orgid)
      
      $ export ACTIVATION_KEY=$(cat ~/.rh_subscription/activation_key)

   2. When using an unsubscribed RHEL system, set the appropriate subscription values by running the following commands:

      $ export ORG_ID=<RHEL_ORGID_VALUE>
      
      $ export ACTIVATION_KEY=<RHEL_ACVTIVATION_KEY>

2. Log in to your IBM Z system and perform the following steps:

   1. Download an s390x RHEL KVM guest image from the Red Hat Customer Portal to your libvirt storage directory to grant libvirt correct access. The default directory is /var/lib/libvirt/images. The image is used to generate the peer pod VM image, which will include the relevant binaries.

   2. Set the IMAGE_URL shell environment variable for the downloaded image by running the following command:

      $ export IMAGE_URL=<location_of_downloaded_KVM_guest_image> 1

      1

      Enter the path of the KVM guest image that you downloaded in the previous step.

   3. Register the guest KVM image by running the following command:

      $ export REGISTER_CMD="subscription-manager register --org=${ORG_ID} \
        --activationkey=${ACTIVATION_KEY}"

   4. Customize the guest KVM image by running the following command:

      $ virt-customize -v -x -a ${IMAGE_URL} --run-command "${REGISTER_CMD}"

   5. Set the checksum of the image by running the following command:

      $ export IMAGE_CHECKSUM=$(sha256sum ${IMAGE_URL} | awk '{ print $1 }')

Procedure

1. Create a YAML file peer-pods-secret.yaml with the following manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt" 1
     LIBVIRT_URI: "<libvirt_gateway_uri>" 2
     LIBVIRT_POOL: "<libvirt_pool>" 3
     LIBVIRT_VOL_NAME: "<libvirt_volume>" 4

   1

   Enter libvirt as the cloud provider.

   2

   Enter the libvirt_gateway_uri value you retrieved.

   3

   Enter the libvirt_pool value you retrieved.

   4

   Enter the libvirt_volume value you retrieved.

2. Create the secret object:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Create a YAML file peer-pods-cm.yaml with the following manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     PROXY_TIMEOUT: "15m"

2. Apply the ConfigMap:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Generate the SSH key:

   $ ssh-keygen -f ./id_rsa -N ""

2. Copy the SSH public key to your KVM host:

   $ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_ADDRESS> 1

   1

   Enter the IP address of your KVM host.

3. Create the secret object:

   $ oc create secret generic ssh-key-secret \
       -n openshift-sandboxed-containers-operator \
       --from-file=id_rsa.pub=./id_rsa.pub \
       --from-file=id_rsa=./id_rsa

4. Delete the SSH keys:

   $ shred –remove id_rsa.pub id_rsa