Deploying OpenShift sandboxed containers

High-level workflow

1. Install the Node Feature Discovery Operator.
2. Create the NodeFeatureDiscovery custom resource (CR).
3. Enable node eligibility checks when you create the Kataconfig CR. You can run node eligibility checks on all worker nodes or on selected nodes.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     checkNodeEligibility: false 

   1

   
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   2

   Copy to Clipboard Toggle word wrap

   1

   Optional: Set`checkNodeEligibility` to true to run node eligibility checks if you have installed the Node Feature Discovery Operator.

   2

   Optional: If you have applied node labels to install OpenShift sandboxed containers on specific nodes, specify the key and value.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata as a runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata is installed on the cluster.

Procedure

1. Obtain the RuntimeClass object by running the following command:

   $ oc describe runtimeclass kata

   Copy to Clipboard Toggle word wrap

2. Update the overhead.podFixed.memory and cpu values and save as the file as runtimeclass.yaml:

   kind: RuntimeClass
   apiVersion: node.k8s.io/v1
   metadata:
     name: kata
   overhead:
     podFixed:
       memory: "500Mi"
       cpu: "500m"

   Copy to Clipboard Toggle word wrap

3. Apply the changes by running the following command:

   $ oc apply -f runtimeclass.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   Copy to Clipboard Toggle word wrap

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

   Copy to Clipboard Toggle word wrap

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

   Copy to Clipboard Toggle word wrap

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

   Copy to Clipboard Toggle word wrap

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION; \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION; \
   done

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
        -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      Copy to Clipboard Toggle word wrap

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.aws.region}') \
        && echo "AWS_REGION: \"$AWS_REGION\""

      Copy to Clipboard Toggle word wrap

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} \
          --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

      Copy to Clipboard Toggle word wrap

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} \
          --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

      Copy to Clipboard Toggle word wrap

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
        --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") \
          && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

      Copy to Clipboard Toggle word wrap

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     PODVM_INSTANCE_TYPE: "t3.medium"
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large"
     PODVM_AMI_ID: "<podvm_ami_id>"
     AWS_REGION: "<aws_region>"
     AWS_SUBNET_ID: "<aws_subnet_id>"
     AWS_VPC_ID: "<aws_vpc_id>"
     AWS_SG_IDS: "<aws_sg_ids>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy to Clipboard Toggle word wrap

   PODVM_INSTANCE_TYPE

   Defines the default instance type that is used if the instance type is not defined in the workload object.

   PODVM_INSTANCE_TYPES

   Specify the allowed instance types, without spaces, for creating the pod. You can define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   PODVM_AMI_ID

   This value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy to Clipboard Toggle word wrap

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy to Clipboard Toggle word wrap

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy to Clipboard Toggle word wrap

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true

   Copy to Clipboard Toggle word wrap

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy to Clipboard Toggle word wrap

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard Toggle word wrap

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy to Clipboard Toggle word wrap

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy to Clipboard Toggle word wrap

2. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy to Clipboard Toggle word wrap

2. Optional: To use a manually defined instance type, add the following annotation with the instance type that you defined in the config map:

   apiVersion: v1
   kind: <object>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.machine_type: <machine_type>
   # ...

   Copy to Clipboard Toggle word wrap

3. Optional: To use an automatic instance type, add the following annotations:

   apiVersion: v1
   kind: <Pod>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
       io.katacontainers.config.hypervisor.default_memory: <memory>
   # ...

   Copy to Clipboard Toggle word wrap

   The workload will run on an automatic instance type based on the amount of memory available.

4. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
        && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

      Copy to Clipboard Toggle word wrap

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Name:name}" --output tsv)

      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
        --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
         && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

      Copy to Clipboard Toggle word wrap

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

      Copy to Clipboard Toggle word wrap

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "{Location:location}" --output tsv) \
        && echo "AZURE_REGION: \"$AZURE_REGION\""

      Copy to Clipboard Toggle word wrap

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5"
     AZURE_SUBNET_ID: "<azure_subnet_id>"
     AZURE_NSG_ID: "<azure_nsg_id>"
     AZURE_IMAGE_ID: ""
     AZURE_REGION: "<azure_region>"
     AZURE_RESOURCE_GROUP: "<azure_resource_group>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy to Clipboard Toggle word wrap

   AZURE_INSTANCE_SIZE

   Defines the default instance size that is used if the instance size is not defined in the workload object.

   AZURE_IMAGE_ID

   Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod VM image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster.

   AZURE_INSTANCE_SIZES

   Specify the allowed instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy to Clipboard Toggle word wrap

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy to Clipboard Toggle word wrap

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy to Clipboard Toggle word wrap

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

   Copy to Clipboard Toggle word wrap

2. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

   Copy to Clipboard Toggle word wrap

3. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true

   Copy to Clipboard Toggle word wrap

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy to Clipboard Toggle word wrap

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard Toggle word wrap

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy to Clipboard Toggle word wrap

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy to Clipboard Toggle word wrap

2. Check the status stanza of the YAML file.

   If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy to Clipboard Toggle word wrap

2. Optional: To use a manually defined instance size, add the following annotation with the instance size that you defined in the config map:

   apiVersion: v1
   kind: <object>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.machine_type: <machine_type>
   # ...

   Copy to Clipboard Toggle word wrap

3. Optional: To use an automatic instance size, add the following annotations:

   apiVersion: v1
   kind: <Pod>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
       io.katacontainers.config.hypervisor.default_memory: <memory>
   # ...

   Copy to Clipboard Toggle word wrap

   The workload will run on an automatic instance size based on the amount of memory available.

4. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.