User guide

Procedure

1. In the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers Operator tile and click Install.
4. On the Install Operator page, select stable from the list of available Update Channel options.

5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

   Note

   Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
7. Click Install.
8. Navigate to Operators → Installed Operators to verify that the Operator is installed.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator.
3. On the KataConfig tab, click Create KataConfig.

4. Enter the following details:

   • Name: Optional: The default name is example-kataconfig.
   • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • checkNodeEligibility: Optional: Select to use the Node Feature Discovery Operator (NFD) to detect node eligibility.

   • kataConfigPoolSelector. Optional: To install kata on selected nodes, add a match expression for the labels on the selected nodes:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the Key field, enter the label key the selector applies to.
     5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the Values area and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes with the kata runtime class.

5. Click Create. The KataConfig CR is created and installs the kata runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

Verification

1. On the KataConfig tab, click the KataConfig CR to view its details.

2. Click the YAML tab to view the status stanza.

   The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata installation. A message appears each time there is an update.

3. Click Reload to refresh the YAML.

   When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata is installed on the cluster.

Procedure

1. In the OpenShift Container Platform web console, navigate to Workloads → workload type, for example, Pods.
2. On the workload type page, click an object to view its details.
3. Click the YAML tab.

4. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.8.0

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.8.0    1.7.0        Succeeded

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     checkNodeEligibility: false 1
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 2

   1

   Optional: Set`checkNodeEligibility` to true to run node eligibility checks.

   2

   Optional: If you have applied node labels to install OpenShift sandboxed containers on specific nodes, specify the key and value.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata as a runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata is installed on the cluster.

Procedure

1. Obtain the RuntimeClass object by running the following command:

   $ oc describe runtimeclass kata

2. Update the overhead.podFixed.memory and cpu values and save as the file as runtimeclass.yaml:

   kind: RuntimeClass
   apiVersion: node.k8s.io/v1
   metadata:
     name: kata
   overhead:
     podFixed:
       memory: "500Mi"
       cpu: "500m"

3. Apply the changes by running the following command:

   $ oc apply -f runtimeclass.yaml

Procedure

1. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. In the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers Operator tile and click Install.
4. On the Install Operator page, select stable from the list of available Update Channel options.

5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

   Note

   Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
7. Click Install.
8. Navigate to Operators → Installed Operators to verify that the Operator is installed.

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION \
   done

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Click the OpenShift sandboxed containers Operator tile.
3. Click the Import icon (+) on the top right corner.

4. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<aws_access_key>" 1
     AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2

   1

   Specify the AWS_ACCESS_KEY_ID value.

   2

   Specify the AWS_SECRET_ACCESS_KEY value.

5. Click Save to apply the changes.
6. Navigate to Workloads → Secrets to verify the peer pods secret.

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

2. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
3. Select the OpenShift sandboxed containers Operator from the list of operators.
4. Click the Import icon (+) in the top right corner.

5. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 1
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<podvm_ami_id>" 3
     AWS_REGION: "<aws_region>" 4
     AWS_SUBNET_ID: "<aws_subnet_id>" 5
     AWS_VPC_ID: "<aws_vpc_id>" 6
     AWS_SG_IDS: "<aws_sg_ids>" 7
     DISABLECVM: "true"

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   3

   Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   4

   Specify the AWS_REGION value you retrieved.

   5

   Specify the AWS_SUBNET_ID value you retrieved.

   6

   Specify the AWS_VPC_ID value you retrieved.

   7

   Specify the AWS_SG_IDS value you retrieved.

6. Click Save to apply the changes.
7. Navigate to Workloads → ConfigMaps to view the new config map.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator.
3. On the KataConfig tab, click Create KataConfig.

4. Enter the following details:

   • Name: Optional: The default name is example-kataconfig.
   • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.

   • kataConfigPoolSelector. Optional: To install kata-remote on selected nodes, add a match expression for the labels on the selected nodes:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the Key field, enter the label key the selector applies to.
     5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the Values area and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes with the kata-remote runtime class.

5. Click Create. The KataConfig CR is created and installs the kata-remote runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

Verification

1. On the KataConfig tab, click the KataConfig CR to view its details.

2. Click the YAML tab to view the status stanza.

   The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata-remote installation. A message appears each time there is an update.

3. Click Reload to refresh the YAML.

   When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata-remote is installed on the cluster.

Procedure

1. Navigate to Workloads → ConfigMaps.
2. Click the provider config map to view its details.
3. Click the YAML tab.

4. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

Procedure

1. In the OpenShift Container Platform web console, navigate to Workloads → workload type, for example, Pods.
2. On the workload type page, click an object to view its details.
3. Click the YAML tab.

4. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

5. Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:

   • To use a manually defined instance type, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1
     # ...

     1

     Specify the instance type that you defined in the config map.

   • To use an automatic instance type, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.

6. Click Save to apply the changes.

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.8.0

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.8.0    1.7.0        Succeeded

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION \
   done

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<aws_access_key>" 1
     AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2

   1

   Specify the AWS_ACCESS_KEY_ID value.

   2

   Specify the AWS_SECRET_ACCESS_KEY value.

2. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PODVM_INSTANCE_TYPE: "t3.medium" 1
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
     PROXY_TIMEOUT: "5m"
     PODVM_AMI_ID: "<podvm_ami_id>" 3
     AWS_REGION: "<aws_region>" 4
     AWS_SUBNET_ID: "<aws_subnet_id>" 5
     AWS_VPC_ID: "<aws_vpc_id>" 6
     AWS_SG_IDS: "<aws_sg_ids>" 7
     DISABLECVM: "true"

   1

   Defines the default instance type that is used when a type is not defined in the workload.

   2

   Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   3

   Optional: By default, this value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   4

   Specify the AWS_REGION value you retrieved.

   5

   Specify the AWS_SUBNET_ID value you retrieved.

   6

   Specify the AWS_VPC_ID value you retrieved.

   7

   Specify the AWS_SG_IDS value you retrieved.

3. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

2. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

2. Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:

   • To use a manually defined instance type, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "t3.medium" 1
     # ...

     1

     Specify the instance type that you defined in the config map.

   • To use an automatic instance type, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.

3. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. In the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers Operator tile and click Install.
4. On the Install Operator page, select stable from the list of available Update Channel options.

5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

   Note

   Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
7. Click Install.
8. Navigate to Operators → Installed Operators to verify that the Operator is installed.

Procedure

1. Retrieve the Azure subscription ID by running the following command:

   $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" \
     -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""

2. Generate the RBAC content by running the following command:

   $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
     --query "{ client_id: appId, client_secret: password, tenant_id: tenant }"

   Example output

   {
     "client_id": `AZURE_CLIENT_ID`,
     "client_secret": `AZURE_CLIENT_SECRET`,
     "tenant_id": `AZURE_TENANT_ID`
   }

3. Record the RBAC output to use in the secret object.
4. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
5. Click the OpenShift sandboxed containers Operator tile.
6. Click the Import icon (+) on the top right corner.

7. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<azure_client_id>" 1
     AZURE_CLIENT_SECRET: "<azure_client_secret>" 2
     AZURE_TENANT_ID: "<azure_tenant_id>" 3
     AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>" 4

   1

   Specify the AZURE_CLIENT_ID value.

   2

   Specify the AZURE_CLIENT_SECRET value.

   3

   Specify the AZURE_TENANT_ID value.

   4

   Specify the AZURE_SUBSCRIPTION_ID value.

8. Click Save to apply the changes.
9. Navigate to Workloads → Secrets to verify the peer pods secret.

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""

2. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
3. Select the OpenShift sandboxed containers Operator from the list of operators.
4. Click the Import icon (+) in the top right corner.

5. In the Import YAML window, paste the following YAML manifest:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2
     AZURE_SUBNET_ID: "<azure_subnet_id>" 3
     AZURE_NSG_ID: "<azure_nsg_id>" 4
     PROXY_TIMEOUT: "5m"
     AZURE_IMAGE_ID: "<azure_image_id>" 5
     AZURE_REGION: "<azure_region>" 6
     AZURE_RESOURCE_GROUP: "<azure_resource_group>" 7
     DISABLECVM: "true"

   1

   This value is the default if an instance size is not defined in the workload.

   2

   Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   3

   Specify the AZURE_SUBNET_ID value that you retrieved.

   4

   Specify the AZURE_NSG_ID value that you retrieved.

   5

   Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.

   6

   Specify the AZURE_REGION value you retrieved.

   7

   Specify the AZURE_RESOURCE_GROUP value you retrieved.

6. Click Save to apply the changes.
7. Navigate to Workloads → ConfigMaps to view the new config map.

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

2. In the OpenShift Container Platform web console, navigate to Workloads → Secrets.
3. On the Secrets page, verify that you are in the openshift-sandboxed-containers-operator project.
4. Click Create and select Key/value secret.
5. In the Secret name field, enter ssh-key-secret.
6. In the Key field, enter id_rsa.pub.
7. In the Value field, paste your public SSH key.
8. Click Create.

9. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator.
3. On the KataConfig tab, click Create KataConfig.

4. Enter the following details:

   • Name: Optional: The default name is example-kataconfig.
   • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.

   • kataConfigPoolSelector. Optional: To install kata-remote on selected nodes, add a match expression for the labels on the selected nodes:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the Key field, enter the label key the selector applies to.
     5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the Values area and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes with the kata-remote runtime class.

5. Click Create. The KataConfig CR is created and installs the kata-remote runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

Verification

1. On the KataConfig tab, click the KataConfig CR to view its details.

2. Click the YAML tab to view the status stanza.

   The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata-remote installation. A message appears each time there is an update.

3. Click Reload to refresh the YAML.

   When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata-remote is installed on the cluster.

Procedure

1. Navigate to Workloads → ConfigMaps.
2. Click the provider config map to view its details.
3. Click the YAML tab.

4. Check the status stanza of the YAML file.

   If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

Procedure

1. In the OpenShift Container Platform web console, navigate to Workloads → workload type, for example, Pods.
2. On the workload type page, click an object to view its details.
3. Click the YAML tab.

4. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

5. Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:

   • To use a manually defined instance size, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "Standard_B2als_v2" 1
     # ...

     1

     Specify the instance size that you defined in the config map.

   • To use an automatic instance size, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.

6. Click Save to apply the changes.

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.8.0

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.8.0    1.7.0        Succeeded

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Retrieve the Azure subscription ID by running the following command:

   $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" \
     -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""

2. Generate the RBAC content by running the following command:

   $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
     --query "{ client_id: appId, client_secret: password, tenant_id: tenant }"

   Example output

   {
     "client_id": `AZURE_CLIENT_ID`,
     "client_secret": `AZURE_CLIENT_SECRET`,
     "tenant_id": `AZURE_TENANT_ID`
   }

3. Record the RBAC output to use in the secret object.

4. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     AZURE_CLIENT_ID: "<azure_client_id>" 1
     AZURE_CLIENT_SECRET: "<azure_client_secret>" 2
     AZURE_TENANT_ID: "<azure_tenant_id>" 3
     AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>" 4

   1

   Specify the AZURE_CLIENT_ID value.

   2

   Specify the AZURE_CLIENT_SECRET value.

   3

   Specify the AZURE_TENANT_ID value.

   4

   Specify the AZURE_SUBSCRIPTION_ID value.

5. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2
     AZURE_SUBNET_ID: "<azure_subnet_id>" 3
     AZURE_NSG_ID: "<azure_nsg_id>" 4
     PROXY_TIMEOUT: "5m"
     AZURE_IMAGE_ID: "<azure_image_id>" 5
     AZURE_REGION: "<azure_region>" 6
     AZURE_RESOURCE_GROUP: "<azure_resource_group>" 7
     DISABLECVM: "true"

   1

   This value is the default if an instance size is not defined in the workload.

   2

   Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   3

   Specify the AZURE_SUBNET_ID value that you retrieved.

   4

   Specify the AZURE_NSG_ID value that you retrieved.

   5

   Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.

   6

   Specify the AZURE_REGION value you retrieved.

   7

   Specify the AZURE_RESOURCE_GROUP value you retrieved.

3. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

2. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

3. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

2. Check the status stanza of the YAML file.

   If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

2. Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:

   • To use a manually defined instance size, add the following annotation:

     apiVersion: v1
     kind: <object>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.machine_type: "Standard_B2als_v2" 1
     # ...

     1

     Specify the instance size that you defined in the config map.

   • To use an automatic instance size, add the following annotations:

     apiVersion: v1
     kind: <Pod>
     metadata:
       annotations:
         io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
         io.katacontainers.config.hypervisor.default_memory: <memory>
     # ...

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.

3. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an rvps-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: rvps-reference-values
     namespace: trustee-operator-system
   data:
     reference-values.json: |
       [ 1
       ]

   1

   Specify the trusted digests for your hardware platform if required. Otherwise, leave it empty.

2. Create the RVPS config map by running the following command:

   $ oc apply -f rvps-configmap.yaml

Procedure

1. Create an attestation-policy.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: attestation-policy
     namespace: trustee-operator-system
   data:
     default.rego: |
       package policy 1
       import future.keywords.every
   
       default allow = false
   
       allow {
         every k, v in input {
             judge_field(k, v)
         }
       }
   
       judge_field(input_key, input_value) {
         has_key(data.reference, input_key)
         reference_value := data.reference[input_key]
         match_value(reference_value, input_value)
       }
   
       judge_field(input_key, input_value) {
         not has_key(data.reference, input_key)
       }
   
       match_value(reference_value, input_value) {
         not is_array(reference_value)
         input_value == reference_value
       }
   
       match_value(reference_value, input_value) {
         is_array(reference_value)
         array_include(reference_value, input_value)
       }
   
       array_include(reference_value_array, input_value) {
         reference_value_array == []
       }
   
       array_include(reference_value_array, input_value) {
         reference_value_array != []
         some i
         reference_value_array[i] == input_value
       }
   
       has_key(m, k) {
         _ = m[k]
       }

   1

   The attestation policy follows the Open Policy Agent specification. In this example, the attestation policy compares the claims provided in the attestation report to the reference values registered in the RVPS database. The attestation process is successful only if all the values match.

2. Create the attestation policy config map by running the following command:

   $ oc apply -f attestation-policy.yaml

Procedure

1. Create a tdx-config.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tdx-config
     namespace: trustee-operator-system
   data:
     sgx_default_qcnl.conf: | \
         {
           "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/",
           "pccs_url": "<pccs_url>" 1
         }

   1

   Specify the PCCS URL, for example, https://localhost:8081/sgx/certification/v4/.

2. Create the TDX config map by running the following command:

   $ oc apply -f tdx-config.yaml

Procedure

1. Create a secret for container image signature verification by running the following command:

   $ oc apply secret generic <type> \ 1
     --from-file=<tag>=./<public_key_file> \ 2
     -n trustee-operator-system

   1

   Specify the KBS secret type, for example, img-sig.

   2

   Specify the secret tag, for example, pub-key, and the public container image signing key.

2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

Procedure

1. Create a security-policy-config.json file according to the following examples:

   • Without signature verification:

     {
       "default": [
       {
         "type": "insecureAcceptAnything"
       }],
       "transports": {}
     }

   • With signature verification:

     {
       "default": [
           {
           "type": "insecureAcceptAnything"
           }
       ],
       "transports": {
           "<transport>": { 1
               "<registry>/<image>": 2
               [
                   {
                       "type": "sigstoreSigned",
                       "keyPath": "kbs:///default/<type>/<tag>" 3
                   }
               ]
           }
       }
     }

     1

     Specify the image repository for transport, for example, "docker":. For more information, see containers-transports 5.

     2

     Specify the container registry and image, for example, "quay.io/my-image".

     3

     Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.

2. Create the security policy by running the following command:

   $ oc apply secret generic security-policy \
     --from-file=osc=./<security-policy-config.json> \
     -n trustee-operator-system

   Do not alter the secret type, security-policy, or the key, osc.

   The security-policy secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

Procedure

1. Create a resourcepolicy-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: resource-policy
     namespace: trustee-operator-system
   data:
     policy.rego: | 1
       package policy 2
       default allow = false
       allow {
         input["tee"] != "sample"
       }

   1

   The name of the resource policy, policy.rego, must match the resource policy defined in the Trustee config map.

   2

   The resource policy follows the Open Policy Agent specification. This example allows the retrieval of all resources when the TEE is not the sample attester.

2. Create the resource policy config map by running the following command:

   $ oc apply -f resourcepolicy-configmap.yaml

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.8.0

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.8.0    1.7.0        Succeeded

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Log in to the KVM host.

2. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   You need the LIBVIRT_POOL value to create the secret for the libvirt provider.

3. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_VOL_NAME=<libvirt_volume>

   You need the LIBVIRT_VOL_NAME value to create the secret for the libvirt provider.

4. Set the path of the default storage pool location, by running the following command:

   $ export LIBVIRT_POOL_DIRECTORY=<target_directory> 1

   1

   To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.

5. Create a libvirt pool by running the following command:

   $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

6. Start the libvirt pool by running the following command:

   $ virsh pool-start $LIBVIRT_POOL

7. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

Procedure

1. Create a Dockerfile.podvm-oci file:

   FROM scratch
   
   ARG PODVM_IMAGE_SRC
   ENV PODVM_IMAGE_PATH="/image/podvm.qcow2"
   
   COPY $PODVM_IMAGE_SRC $PODVM_IMAGE_PATH

2. Build a container with the pod VM QCOW2 image by running the following command:

   $ docker build -t podvm-libvirt \
     --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 1
     --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 2
     -f Dockerfile.podvm-oci .

   1

   Specify the QCOW2 image source on the host.

   2

   Optional: Specify the path of the QCOW2 image if you do not use the default, /image/podvm.qcow2.

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_URI: "<libvirt_gateway_uri>" 1
     LIBVIRT_POOL: "<libvirt_pool>" 2
     LIBVIRT_VOL_NAME: "<libvirt_volume>" 3
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 4

   1

   Specify the libvirt URI.

   2

   Specify the libvirt pool.

   3

   Specify the libvirt volume name.

   4

   Specify the Red Hat offline token, which is required for the Operator-built image.

2. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     DISABLECVM: "true"

2. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Create a libvirt-podvm-image-cm.yaml manifest according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: libvirt-podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     PODVM_DISTRO: "rhel"
     CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor"
     CAA_REF: "<cloud_api_adaptor_version>" 1
     DOWNLOAD_SOURCES: "no"
     CONFIDENTIAL_COMPUTE_ENABLED: "yes"
     UPDATE_PEERPODS_CM: "yes"
     ORG_ID: "<rhel_organization_id>"
     ACTIVATION_KEY: "<rhel_activation_key>" 2
     IMAGE_NAME: "<podvm_libvirt_image>"
     PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 3
     SE_BOOT: "true" 4
     BASE_OS_VERSION: "<rhel_image_os_version>" 5

   1

   Specify the latest version of the Cloud API Adaptor source.

   2

   Specify your RHEL activation key.

   3

   Optional: Specify the following values if you created a container image:

   • image_repo_url: Container registry URL.
   • image_tag: Image tag.
   • image_path: Image path. Default: /image/podvm.qcow2.

   4

   SE_BOOT: "true" enables IBM Secure Execution for an Operator-built image. Set to false if you created a container image.

   5

   Specify the RHEL image operating system version. IBM Z® Secure Execution supports RHEL 9.4 and later versions.

2. Create the config map by running the following command:

   $ oc apply -f libvirt-podvm-image-cm.yaml

   The libvirt pod VM image config map is created for your libvirt provider.

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

2. Copy the public SSH key to your KVM host:

   $ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP>

3. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

4. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   
   BinaryData
   ====
   Events: <none>

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an rvps-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: rvps-reference-values
     namespace: trustee-operator-system
   data:
     reference-values.json: |
       [ 1
       ]

   1

   Leave this value empty.

2. Create the RVPS config map by running the following command:

   $ oc apply -f rvps-configmap.yaml

Procedure

1. Create an attestation-policy.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: attestation-policy
     namespace: trustee-operator-system
   data:
     default.rego: | 1
       package policy
       import rego.v1
       default allow = false
       converted_version := sprintf("%v", [input["se.version"]])
       allow if {
           input["se.attestation_phkh"] == "<se.attestation_phkh>" 2
           input["se.image_phkh"] == "<se.image_phkh>"
           input["se.tag"] == "<se.tag>"
           converted_version == "256"
       }

   1

   Do not modify the policy name.

   2

   Specify the attestation policy fields you obtained by running the se_parse_hdr.py script.

2. Create the attestation policy config map by running the following command:

   $ oc apply -f attestation-policy.yaml

Procedure

1. Create a secret for container image signature verification by running the following command:

   $ oc apply secret generic <type> \ 1
     --from-file=<tag>=./<public_key_file> \ 2
     -n trustee-operator-system

   1

   Specify the KBS secret type, for example, img-sig.

   2

   Specify the secret tag, for example, pub-key, and the public container image signing key.

2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

Procedure

1. Create a security-policy-config.json file according to the following examples:

   • Without signature verification:

     {
       "default": [
       {
         "type": "insecureAcceptAnything"
       }],
       "transports": {}
     }

   • With signature verification:

     {
       "default": [
           {
           "type": "insecureAcceptAnything"
           }
       ],
       "transports": {
           "<transport>": { 1
               "<registry>/<image>": 2
               [
                   {
                       "type": "sigstoreSigned",
                       "keyPath": "kbs:///default/<type>/<tag>" 3
                   }
               ]
           }
       }
     }

     1

     Specify the image repository for transport, for example, "docker":. For more information, see containers-transports 5.

     2

     Specify the container registry and image, for example, "quay.io/my-image".

     3

     Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.

2. Create the security policy by running the following command:

   $ oc apply secret generic security-policy \
     --from-file=osc=./<security-policy-config.json> \
     -n trustee-operator-system

   Do not alter the secret type, security-policy, or the key, osc.

   The security-policy secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

Procedure

1. Create a resourcepolicy-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: resource-policy
     namespace: trustee-operator-system
   data:
     policy.rego: | 1
       package policy 2
       path := split(data["resource-path"], "/")
       default allow = false
       allow {
         count(path) == 3
         input["tee"] == "se"
       }

   1

   The name of the resource policy, policy.rego, must match the resource policy defined in the Trustee config map.

   2

   The resource policy follows the Open Policy Agent specification. This example allows the retrieval of all resources when the TEE is not the sample attester.

2. Create the resource policy config map by running the following command:

   $ oc apply -f resourcepolicy-configmap.yaml

Procedure

1. Change the logLevel field in your existing KataConfig CR to debug:

   $ oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}'

2. Monitor the kata-oc machine config pool until the value of UPDATED is True, indicating that all worker nodes are updated:

   $ oc get mcp kata-oc

   Example output

   NAME     CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
   kata-oc  rendered-kata-oc-169   False    True      False     3             1                  1                    0                     9h

Verification

1. Start a debug session with a node in the machine config pool:

   $ oc debug node/<node_name>

2. Change the root directory to /host:

   # chroot /host

3. Verify the changes in the crio.conf file:

   # crio config | egrep 'log_level

   Example output

   log_level = "debug"