User guide

Procedure

1. In the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers Operator tile and click Install.
4. On the Install Operator page, select stable from the list of available Update Channel options.

5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

   Note

   Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
7. Click Install.
8. Navigate to Operators → Installed Operators to verify that the Operator is installed.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator.
3. On the KataConfig tab, click Create KataConfig.

4. Enter the following details:

   • Name: Optional: The default name is example-kataconfig.
   • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • checkNodeEligibility: Optional: Select to use the Node Feature Discovery Operator (NFD) to detect node eligibility.

   • kataConfigPoolSelector. Optional: To install kata on selected nodes, add a match expression for the labels on the selected nodes:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the Key field, enter the label key the selector applies to.
     5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the Values area and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes with the kata runtime class.

5. Click Create. The KataConfig CR is created and installs the kata runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

Verification

1. On the KataConfig tab, click the KataConfig CR to view its details.

2. Click the YAML tab to view the status stanza.

   The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata installation. A message appears each time there is an update.

3. Click Reload to refresh the YAML.

   When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata is installed on the cluster.

Procedure

1. In the OpenShift Container Platform web console, navigate to Workloads → workload type, for example, Pods.
2. On the workload type page, click an object to view its details.
3. Click the YAML tab.

4. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.9.0

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.9.0    1.8.1        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     checkNodeEligibility: false 

   1

   
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   2

   Copy to Clipboard Toggle word wrap

   1

   Optional: Set`checkNodeEligibility` to true to run node eligibility checks.

   2

   Optional: If you have applied node labels to install OpenShift sandboxed containers on specific nodes, specify the key and value.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata as a runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata is installed on the cluster.

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

   Copy to Clipboard Toggle word wrap

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <value> with the limit you want to define.

Procedure

1. Obtain the RuntimeClass object by running the following command:

   $ oc describe runtimeclass kata

   Copy to Clipboard Toggle word wrap

2. Update the overhead.podFixed.memory and cpu values and save as the file as runtimeclass.yaml:

   kind: RuntimeClass
   apiVersion: node.k8s.io/v1
   metadata:
     name: kata
   overhead:
     podFixed:
       memory: "500Mi"
       cpu: "500m"

   Copy to Clipboard Toggle word wrap

3. Apply the changes by running the following command:

   $ oc apply -f runtimeclass.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. In the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers Operator tile and click Install.
4. On the Install Operator page, select stable from the list of available Update Channel options.

5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

   Note

   Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
7. Click Install.
8. Navigate to Operators → Installed Operators to verify that the Operator is installed.

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

   Copy to Clipboard Toggle word wrap

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

   Copy to Clipboard Toggle word wrap

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

   Copy to Clipboard Toggle word wrap

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION \
   done

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Click the OpenShift sandboxed containers Operator tile.
3. Click the Import icon (+) on the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<aws_access_key>" 

   1

   
     AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the AWS_ACCESS_KEY_ID value.

   2

   Specify the AWS_SECRET_ACCESS_KEY value.

5. Click Save to apply the changes.
6. Navigate to Workloads → Secrets to verify the peer pods secret.

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      Copy to Clipboard Toggle word wrap

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

      Copy to Clipboard Toggle word wrap

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

      Copy to Clipboard Toggle word wrap

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

      Copy to Clipboard Toggle word wrap

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

      Copy to Clipboard Toggle word wrap
2. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
3. Select the OpenShift sandboxed containers Operator from the list of operators.
4. Click the Import icon (+) in the top right corner.

5. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 

   1

   
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 

   2

   
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<podvm_ami_id>" 

   3

   
     AWS_REGION: "<aws_region>" 

   4

   
     AWS_SUBNET_ID: "<aws_subnet_id>" 

   5

   
     AWS_VPC_ID: "<aws_vpc_id>" 

   6

   
     AWS_SG_IDS: "<aws_sg_ids>" 

   7

   
     PEERPODS_LIMIT_PER_NODE: "10" 

   8

   
     TAGS: "key1=value1,key2=value2" 

   9

   
     DISABLECVM: "true"

   Copy to Clipboard Toggle word wrap

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Specify the instance types, without spaces, for creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   3

   Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   4

   Specify the AWS_REGION value you retrieved.

   5

   Specify the AWS_SUBNET_ID value you retrieved.

   6

   Specify the AWS_VPC_ID value you retrieved.

   7

   Specify the AWS_SG_IDS value you retrieved.

   8

   Specify the maximum number of peer pods that can be created per node. The default value is 10.

   9

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

6. Click Save to apply the changes.
7. Navigate to Workloads → ConfigMaps to view the new config map.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. Click the Import icon (+) in the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>" 

   1

   
   spec:
     runtimeClassName: kata-remote 

   2

   
     containers:
     - name: <example_container> 

   3

   
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap
5. Click Save to apply the changes.

Procedure

1. Create a policy.rego file with your custom policy. The following example shows all configurable APIs, with exec and log enabled for demonstration:

   package agent_policy
   
   import future.keywords.in
   import input
   
   default CopyFileRequest := false
   default CreateContainerRequest := false
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true  # Enabled to allow exec API
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default OnlineCPUMemRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := true   # Enabled to allow log API
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StatsContainerRequest := true
   default TtyWinResizeRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := false

   Copy to Clipboard Toggle word wrap

   This policy enables the exec (ExecProcessRequest) and log (ReadStreamRequest) APIs. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy to Clipboard Toggle word wrap

   Save the output for use in the yaml file.

3. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
4. Select the OpenShift sandboxed containers Operator from the list of operators.
5. Click the Import icon (+) in the top right corner.

6. In the Import YAML window, paste the following YAML manifest and add the Base64-encoded policy to it:

   apiVersion: v1
   kind: Pod
   metadata:
     name: <pod_name>
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap
7. Click Save to apply the changes.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator.
3. On the KataConfig tab, click Create KataConfig.

4. Enter the following details:

   • Name: Optional: The default name is example-kataconfig.
   • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.

   • kataConfigPoolSelector. Optional: To install kata-remote on selected nodes, add a match expression for the labels on the selected nodes:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the Key field, enter the label key the selector applies to.
     5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the Values area and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes with the kata-remote runtime class.

5. Click Create. The KataConfig CR is created and installs the kata-remote runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

Verification

1. On the KataConfig tab, click the KataConfig CR to view its details.

2. Click the YAML tab to view the status stanza.

   The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata-remote installation. A message appears each time there is an update.

3. Click Reload to refresh the YAML.

   When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata-remote is installed on the cluster.

Procedure

1. Navigate to Workloads → ConfigMaps.
2. Click the provider config map to view its details.
3. Click the YAML tab.

4. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform web console, navigate to Workloads → workload type, for example, Pods.
2. On the workload type page, click an object to view its details.
3. Click the YAML tab.

4. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy to Clipboard Toggle word wrap

5. Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:

   • To use a manually defined instance type, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "t3.medium" 

     1

     
     # ...

     Copy to Clipboard Toggle word wrap

     1 1

     Specify the instance type that you defined in the config map.

   • To use an automatic instance type, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Copy to Clipboard Toggle word wrap

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.

6. Click Save to apply the changes.

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.9.0

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.9.0    1.8.1        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

   Copy to Clipboard Toggle word wrap

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

   Copy to Clipboard Toggle word wrap

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

   Copy to Clipboard Toggle word wrap

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION \
   done

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<aws_access_key>" 

   1

   
     AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the AWS_ACCESS_KEY_ID value.

   2

   Specify the AWS_SECRET_ACCESS_KEY value.

2. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      Copy to Clipboard Toggle word wrap

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

      Copy to Clipboard Toggle word wrap

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

      Copy to Clipboard Toggle word wrap

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

      Copy to Clipboard Toggle word wrap

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

      Copy to Clipboard Toggle word wrap

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 

   1

   
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 

   2

   
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<podvm_ami_id>" 

   3

   
     AWS_REGION: "<aws_region>" 

   4

   
     AWS_SUBNET_ID: "<aws_subnet_id>" 

   5

   
     AWS_VPC_ID: "<aws_vpc_id>" 

   6

   
     AWS_SG_IDS: "<aws_sg_ids>" 

   7

   
     PEERPODS_LIMIT_PER_NODE: "10" 

   8

   
     TAGS: "key1=value1,key2=value2" 

   9

   
     DISABLECVM: "true"

   Copy to Clipboard Toggle word wrap

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Specify the instance types, without spaces, for creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   3

   Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   4

   Specify the AWS_REGION value you retrieved.

   5

   Specify the AWS_SUBNET_ID value you retrieved.

   6

   Specify the AWS_VPC_ID value you retrieved.

   7

   Specify the AWS_SG_IDS value you retrieved.

   8

   Specify the maximum number of peer pods that can be created per node. The default value is 10.

   9

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

3. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the pod manifest by adding the io.katacontainers.config.hypervisor.image annotation and save it in a pod-manifest.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>" 

   1

   
   spec:
     runtimeClassName: kata-remote 

   2

   
     containers:
     - name: <example_container> 

   3

   
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap

   1

   Specify the custom peer pod image ID.

   2

   Ensure that the runtimeClassName field is set to kata-remote to create a peer pod.

   3

   Specify the container name.

2. Create the pod by running the following command:

   $ oc apply -f pod-manifest.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a policy.rego file with your custom policy. The following example shows all configurable APIs, with exec and log enabled for demonstration:

   package agent_policy
   
   import future.keywords.in
   import input
   
   default CopyFileRequest := false
   default CreateContainerRequest := false
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true  # Enabled to allow exec API
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default OnlineCPUMemRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := true   # Enabled to allow log API
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StatsContainerRequest := true
   default TtyWinResizeRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := false

   Copy to Clipboard Toggle word wrap

   This policy enables the exec (ExecProcessRequest) and log (ReadStreamRequest) APIs. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy to Clipboard Toggle word wrap

   Save the output for use in the yaml file.

3. Add the Base64-encoded policy to a my-pod.yaml pod specification file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: <pod_name>
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

4. Apply the pod manifest by running the following command:

   $ oc apply -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard Toggle word wrap

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

   Copy to Clipboard Toggle word wrap

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <value> with the limit you want to define.

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy to Clipboard Toggle word wrap

2. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy to Clipboard Toggle word wrap

2. Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:

   • To use a manually defined instance type, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "t3.medium" 

     1

     
     # ...

     Copy to Clipboard Toggle word wrap

     1

     Specify the instance type that you defined in the config map.

   • To use an automatic instance type, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Copy to Clipboard Toggle word wrap

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.

3. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Set the AZURE_RESOURCE_GROUP environment variable by running the following command:

   $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
       -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

   Copy to Clipboard Toggle word wrap

2. Set the AZURE_REGION environment variable by running the following command:

   $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
       --query "{Location:location}" --output tsv) && \
       echo "AZURE_REGION: \"$AZURE_REGION\""

   Copy to Clipboard Toggle word wrap

3. Set the AZURE_VNET_NAME environment variable by running the following command:

   $ AZURE_VNET_NAME=$(az network vnet list \
       -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)

   Copy to Clipboard Toggle word wrap

4. Set the AZURE_SUBNET_ID environment variable by running the following command:

   $ AZURE_SUBNET_ID=$(az network vnet subnet list \
       --resource-group "${AZURE_RESOURCE_GROUP}" \
       --vnet-name "${AZURE_VNET_NAME}" --query "[].{Id:id} \
       | [? contains(Id, 'worker')]" --output tsv)

   Copy to Clipboard Toggle word wrap

5. Set the NAT gateway environment variables for the peer pod subnet by running the following commands:

   $ export PEERPOD_NAT_GW=peerpod-nat-gw

   Copy to Clipboard Toggle word wrap

   $ export PEERPOD_NAT_GW_IP=peerpod-nat-gw-ip

   Copy to Clipboard Toggle word wrap

6. Create a public IP address for the NAT gateway by running the following command:

   $ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
       -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}" --sku Standard

   Copy to Clipboard Toggle word wrap

7. Create the NAT gateway and associate it with the public IP address by running the following command:

   $ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
       -l "${AZURE_REGION}" --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
       -n "${PEERPOD_NAT_GW}"

   Copy to Clipboard Toggle word wrap

8. Update the VNet subnet to use the NAT gateway by running the following command:

   $ az network vnet subnet update --nat-gateway "${PEERPOD_NAT_GW}" \
       --ids "${AZURE_SUBNET_ID}"

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the environmental variables for the peer pod network:

   1. Set the peer pod VNet environment variables by running the following commands:

      $ export PEERPOD_VNET_NAME="${PEERPOD_VNET_NAME:-peerpod-vnet}"

      Copy to Clipboard Toggle word wrap

      $ export PEERPOD_VNET_CIDR="${PEERPOD_VNET_CIDR:-192.168.0.0/16}"

      Copy to Clipboard Toggle word wrap

   2. Set the peer pod subnet environment variables by running the following commands:

      $ export PEERPOD_SUBNET_NAME="${PEERPOD_SUBNET_NAME:-peerpod-subnet}"

      Copy to Clipboard Toggle word wrap

      $ export PEERPOD_SUBNET_CIDR="${PEERPOD_SUBNET_CIDR:-192.168.0.0/16}"

      Copy to Clipboard Toggle word wrap

2. Set the environmental variables for Azure:

   $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
       -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

   Copy to Clipboard Toggle word wrap

   $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
       --query "{Location:location}" --output tsv) && \
       echo "AZURE_REGION: \"$AZURE_REGION\""

   Copy to Clipboard Toggle word wrap

   $ AZURE_VNET_NAME=$(az network vnet list \
       -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)

   Copy to Clipboard Toggle word wrap

3. Set the peer pod NAT gateway environment variables by running the following commands:

   $ export PEERPOD_NAT_GW="${PEERPOD_NAT_GW:-peerpod-nat-gw}"

   Copy to Clipboard Toggle word wrap

   $ export PEERPOD_NAT_GW_IP="${PEERPOD_NAT_PUBLIC_IP:-peerpod-nat-gw-ip}"

   Copy to Clipboard Toggle word wrap

4. Configure the VNET:

   1. Create the peer pod VNet by running the following command:

      $ az network vnet create --resource-group "${AZURE_RESOURCE_GROUP}" \
          --name "${PEERPOD_VNET_NAME}" \
          --address-prefixes "${PEERPOD_VNET_CIDR}"

      Copy to Clipboard Toggle word wrap

   2. Create a public IP address for the peer pod VNet by running the following command:

      $ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
          -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}"

      Copy to Clipboard Toggle word wrap

   3. Create a NAT gateway for the peer pod VNet by running the following command:

      $ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
          -l "${AZURE_REGION}" \
          --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
          -n "${PEERPOD_NAT_GW}"

      Copy to Clipboard Toggle word wrap

   4. Create a subnet in the peer pod VNet and attach the NAT gateway by running the following command:

      $ az network vnet subnet create \
          --resource-group "${AZURE_RESOURCE_GROUP}" \
          --vnet-name "${PEERPOD_VNET_NAME}" \
          --name "${PEERPOD_SUBNET_NAME}" \
          --address-prefixes "${PEERPOD_SUBNET_CIDR}" \
          --nat-gateway "${PEERPOD_NAT_GW}"

      Copy to Clipboard Toggle word wrap

5. Configure the virtual network peering connection:

   1. Create the peering connection by running the following command:

      $ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-azure-vnet-to-peerpod-vnet \
          --vnet-name "${AZURE_VNET_NAME}" \
          --remote-vnet "${PEERPOD_VNET_NAME}" --allow-vnet-access \
          --allow-forwarded-traffic

      Copy to Clipboard Toggle word wrap

   2. Sync the peering connection by running the following command:

      $ az network vnet peering sync -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-azure-vnet-to-peerpod-vnet \
          --vnet-name "${AZURE_VNET_NAME}"

      Copy to Clipboard Toggle word wrap

   3. Complete the peering connection by running the following command:

      $ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-peerpod-vnet-to-azure-vnet \
          --vnet-name "${PEERPOD_VNET_NAME}" \
          --remote-vnet "${AZURE_VNET_NAME}" --allow-vnet-access \
          --allow-forwarded-traffic

      Copy to Clipboard Toggle word wrap