Chapter 3. Deploying OpenShift sandboxed containers workloads using peer pods
You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI (oc
). Before installing the OpenShift sandboxed containers Operator, you must prepare your OpenShift Container Platform cluster.
3.1. Prerequisites
Before you install OpenShift sandboxed containers and enable peer pods, you must meet the following requirements:
- You have OpenShift Container Platform 4.15 installed on AWS or Azure.
For installations on IBM Z or IBM® LinuxONE, you must have OpenShift Container Platform 4.14 or later installed.
ImportantDeploying OpenShift sandboxed containers workloads on IBM Z using peer pods is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
-
You have installed the OpenShift CLI (
oc
). - You have access to the cluster as a user with the cluster-admin role.
3.1.1. About peer pod resource requirements in OpenShift sandboxed containers
A peer pod uses resources in two locations:
-
The worker node. The worker node stores metadata, Kata shim resources (
containerd-shim-kata-v2
), remote-hypervisor resources (cloud-api-adaptor
), and the tunnel setup between the worker nodes and the peer-pod virtual machine (VM). - The cloud instance. This is the actual peer-pod VM running in the cloud.
The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote
) definition used for creating peer pods.
The total number of peer-pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit
attribute in the peerpodConfig
custom resource (CR). The peerpodConfig
CR, named peerpodconfig-openshift
, is created when you create the kataConfig
CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator
namespace.
The following peerpodConfig
CR example displays the default spec
values:
apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
name: peerpodconfig-openshift
namespace: openshift-sandboxed-containers-operator
spec:
cloudSecretName: peer-pods-secret
configMapName: peer-pods-cm
limit: "10" 1
nodeSelector:
node-role.kubernetes.io/kata-oc: ""
- 1
- The default limit is 10 VMs per node.
The extended resource is named kata.peerpods.io/vm
, and enables the Kubernetes scheduler to handle capacity tracking and accounting.
You can edit the limit per node based on the requirements for your environment. See Modifying the VM limit per node in peer pods for more information.
A mutating webhook adds the extended resource kata.peerpods.io/vm
to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer-pod is only scheduled when resources are available.
The mutating webhook modifies a Kubernetes pod as follows:
-
The mutating webhook checks the pod for the expected
RuntimeClassName
value, specified in theTARGET_RUNTIME_CLASS
environment variable. If the value in the pod specification does not match the value in theTARGET_RUNTIME_CLASS
, the webhook exits without modifying the pod. If the
RuntimeClassName
values match, the webhook makes the following changes to the pod spec:-
The webhook removes every resource specification from the
resources
field of all containers and init containers in the pod. -
The webhook adds the extended resource (
kata.peerpods.io/vm
) to the spec by modifying the resources field of the first container in the pod. The extended resourcekata.peerpods.io/vm
is used by the Kubernetes scheduler for accounting purposes.
-
The webhook removes every resource specification from the
The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer-pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.
As a best practice, define a cluster-wide policy to only allow peer-pod creation in specific namespaces.
3.1.1.1. Modifying the peer pod VM limit per node
You can change the limit of peer pod VMs per node by editing the peerpodConfig
custom resource (CR).
Procedure
Check the current limit by running the following command:
$ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}'
Modify the
limit
attribute of thepeerpodConfig
CR by running the following command:$ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1
- 1
- Replace <value> with the limit you want to define.
3.1.2. Prerequisites for peer-pods using AWS
If you are using AWS to create peer pods, you must ensure the following requirements:
- Your OpenShift Container Platform cluster must be installed on AWS, with at least one worker node.
-
You have access to
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
credentials. These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster. - You must have the AWS CLI tool installed and configured.
You must enable internal cluster communication on ports 15150 and 9000.
You can enable these ports in either the AWS web console or using the CLI.
3.1.2.1. Enable ports 15150 and 9000 for AWS
Procedure
Retrieve the Instance ID:
$ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
Retrieve the AWS region:
$ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
Retrieve the security groups:
$ SG=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region ${AWS_REGION})
Authorize the peer-pods shim and to access kata-agent communication. Run the following command:
$ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 15150 --source-group ${SG} --region ${AWS_REGION}
Set up the peer-pods tunnel. Run the following command:
$ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 9000 --source-group ${SG} --region ${AWS_REGION}
The ports are now enabled.
3.1.3. Prerequisites for peer-pods using Azure
If you are using Microsoft Azure to create peer pods, you must ensure the following requirements:
- Your OpenShift Container Platform cluster must be installed on Azure, with at least one worker node.
You have access to the following credentials and subscription details:
-
AZURE_SUBSCRIPTION_ID
-
AZURE_CLIENT_ID
-
AZURE_CLIENT_SECRET
-
AZURE_TENANT_ID
These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster.
-
- You must have the Azure CLI tool installed and configured.
3.1.4. Prerequisites for peer pods using IBM Z or IBM(R) LinuxONE with RHEL KVM
You can install the OpenShift sandboxed containers Operator on IBM Z using the OpenShift CLI (oc
).
While this document refers only to IBM Z, all information in it also applies to IBM® LinuxONE.
If you are using IBM Z to create peer pods, you must ensure the following requirements:
- Your OpenShift Container Platform cluster must be installed on IBM Z, with at least one compute node.
You have installed the following tools on the IBM Z KVM host
- libvirt
- podman
- git
- tar
- virt-customize
-
You must have the
oc
CLI tool installed and configured.
If you want to test peer pods in a non-production environment, the following requirements must be met:
- Your OpenShift Container Platform cluster must be version 4.14 or later.
- You have a multi-node OpenShift Container Platform cluster with three control nodes and two compute nodes set up.
- Cluster nodes and peer pods must run on a single IBM Z KVM host logical partition (LPAR).
- Your cluster setup ensures that peer pod virtual machines (VMs) and cluster nodes are part of the same subnet, allowing connection between node and peer pod VM.
3.2. Deploying OpenShift sandboxed containers workloads using peer pods with the web console
You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create a secret object, the VM image, and a peer-pod ConfigMap. The secret object and ConfigMap are unique, depending on your cloud provider. Finally, you must create the KataConfig
custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata-remote
as the runtimeClassName
to the workload YAML file.
3.2.1. Installing the OpenShift sandboxed containers Operator using the web console
You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console.
Prerequisites
- You have OpenShift Container Platform 4.15 installed.
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
OperatorHub. -
In the Filter by keyword field, type
OpenShift sandboxed containers
. - Select the OpenShift sandboxed containers tile.
- Read the information about the Operator and click Install.
On the Install Operator page:
- Select stable from the list of available Update Channel options.
Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory
openshift-sandboxed-containers-operator
namespace. If this namespace does not yet exist, it is automatically created.NoteAttempting to install the OpenShift sandboxed containers Operator in a namespace other than
openshift-sandboxed-containers-operator
causes the installation to fail.- Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
- Click Install.
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.
3.2.2. Configuring peer-pod parameters for AWS using the web console
You must create a secret object and a ConfigMap
to deploy OpenShift sandboxed containers using peer pods on AWS.
After creating the secret object, you must still create the KataConfig
custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
3.2.2.1. Creating a secret object for AWS using the web console
Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.
When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for AWS using the CLI.
In addition, in the AWS web console, you must generate and save the following values:
-
AWS_ACCESS_KEY_ID
-
AWS_SECRET_ACCESS_KEY
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<enter value>" 1 AWS_SECRET_ACCESS_KEY: "<enter value>" 2 AWS_REGION: "<enter value>" 3 AWS_SUBNET_ID: "<enter value>" 4 AWS_VPC_ID: "<enter value>" 5 AWS_SG_IDS: "<enter value>" 6
- 1
- Enter the
AWS_ACCESS_KEY_ID
value you prepared before you began. - 2
- Enter the
AWS_SECRET_ACCESS_KEY
value you prepared before you began. - 3
- Enter the
AWS_REGION
value you retrieved. - 4
- Enter the
AWS_SUBNET_ID
value you retrieved. - 5
- Enter the
AWS_VPC_ID
value you retrieved. - 6
- Enter the
AWS_SG_IDS
value you retrieved.
- Click Create.
The secret object is created. You can see it listed under Workloads
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the secret, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.2.2.2. Creating a peer-pod ConfigMap for AWS using the web console
You can create a peer-pod ConfigMap
for AWS using the web console.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m"
- Click Create.
The ConfigMap
object is created. You can see it listed under Workloads
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
Once you create the KataConfig
CR, you can run OpenShift sandboxed containers using peer pods on AWS.
3.2.3. Configuring peer-pod parameters for Azure using the web console
You must create a secret object and a ConfigMap
in order to deploy OpenShift sandboxed containers using peer pods on Microsoft Azure.
After creating the secret object, you must still create the KataConfig
custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
3.2.3.1. Creating a secret object for Azure using the web console
Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.
When creating a secret object for Azure, you must set specific environment values. You can retrieve these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for Azure using the CLI.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AZURE_CLIENT_ID: "<enter value>" 1 AZURE_CLIENT_SECRET: "<enter value>" 2 AZURE_TENANT_ID: "<enter value>" 3 AZURE_SUBSCRIPTION_ID: "<enter value>" 4 AZURE_REGION: "<enter value>" 5 AZURE_RESOURCE_GROUP: "<enter value>" 6
- 1
- Enter the
AZURE_CLIENT_ID
value you generated in the RBAC file. - 2
- Enter the
AZURE_CLIENT_SECRET
value you generated in the RBAC file. - 3
- Enter the
AZURE_TENANT_ID
value you generated in the RBAC file. - 4
- Enter the
AZURE_SUBSCRIPTION_ID
value you retrieved. - 5
- Enter the
AZURE_REGION
value you retrieved. - 6
- Enter the
AZURE_RESOURCE_GROUP
value you retrieved.
- Click Create.
The secret object is created. You can see it listed under Workloads
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the secret, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.2.3.2. Creating a peer-pod ConfigMap for Azure using the web console
You can create a peer-pod ConfigMap
for Azure using the web console.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "azure" VXLAN_PORT: "9000" AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1 AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2 AZURE_SUBNET_ID: "<enter value>" 3 AZURE_NSG_ID: "<enter value>" 4 PROXY_TIMEOUT: "5m" DISABLECVM: "true"
- 1
- Defines the default instance size that is used when an instance is not defined in the workload.
- 2
- List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
- 3
- Enter the
AZURE_SUBNET_ID
value that you retrieved. - 4
- Enter the
AZURE_NSG_ID
value that you retrieved.
- Click Create.
The ConfigMap`
object is created. You can see it listed under Workloads
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.2.3.3. Creating an SSH key secret object for Azure using the web console
You must create an SSH key secret object to use peer pods with Azure. If you do not already have an SSH key to create the object, you must generate one using the CLI. Fo more information, see
Procedure
-
From the Administrator perspective in the web console, navigate to Workloads
Secrets. - In the Secrets window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
- Click Create and select Key/value secret from the list.
-
In the Secret name field, enter
ssh-key-secret
. -
In the Key field, enter
id_rsa.pub
. - In the Value field, paste your public SSH key.
- Click Create.
The SSH key secret object is created. You can see it listed under Workloads
Once you create the KataConfig
CR, you can run OpenShift sandboxed containers using peer pods on Azure.
3.2.4. Creating the KataConfig custom resource in the web console
You must create one KataConfig
custom resource (CR) to enable installing kata-remote
as a RuntimeClass
on your cluster nodes.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote
as a RuntimeClass
only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig
CR when you create it.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- In the KataConfig tab, click Create KataConfig.
In the Create KataConfig page, enter the following details:
-
Name: Enter a name for the
KataConfig
resource. By default, the name is defined asexample-kataconfig
. -
Labels (Optional): Enter any relevant, identifying attributes to the
KataConfig
resource. Each label represents a key-value pair. -
checkNodeEligibility
(Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to runkata
as aRuntimeClass
. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers". -
enablePeerPods
(For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment. kataConfigPoolSelector
: By default,kata-remote
is installed as aRuntimeClass
on all nodes. If you want to installkata-remote
as aRuntimeClass
only on selected nodes, you must add a matchExpression:-
Expand the
kataConfigPoolSelector
area. -
In the
kataConfigPoolSelector
, expand matchExpressions. This is a list of label selector requirements. - Click Add matchExpressions.
- In the key field, add the label key the selector applies to.
-
In the operator field, add the key’s relationship to the label values. Valid operators are
In
,NotIn
,Exists
, andDoesNotExist
. - Expand the values area, and then click Add value.
-
In the Value field, enter
true
orfalse
for key label value.
-
Expand the
-
logLevel
: Define the level of log data retrieved for nodes runningkata-remote
as aRuntimeClass
. For more information, see "Collecting OpenShift sandboxed containers data".
-
Name: Enter a name for the
- Click Create.
The new KataConfig
CR is created and begins to install kata-remote
as a RuntimeClass
on the worker nodes. Wait for the installation to complete and the worker nodes to reboot before continuing to the next step.
When you run the CR, the VM image is created. The image creation is done through the cloud provider and it may use additional resources.
OpenShift sandboxed containers installs kata-remote
only as a secondary, optional runtime on the cluster and not as the primary runtime.
Verification
-
In the KataConfig tab, select the new
KataConfig
CR. - In the KataConfig page, select the YAML tab.
Monitor the status field.
A message appears each time there is an update. Click Reload to view the updated
KataConfig
CR.The status field has conditions and kataNodes subfields. The kataNodes subfield is a collection of node lists, each of which lists nodes in a particular state of
kata
installation.When all workers under kataNodes are listed as
installed
, and the conditionInProgress
isFalse
without specifying a reason, this indicates thatkata
is installed on the cluster. For more information, see "Installation and uninstall transitions".
3.2.5. Deploying a workload with peer pods in a sandboxed container using the web console
OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.
To deploy a pod-templated workload using peer pods in a sandboxed container, you must manually add kata-remote
as the runtimeClassName
to the workload YAML file.
You must also define whether the workload should be deployed using a default instance size or type which you previously defined in the ConfigMap
by adding an annotation to the YAML file. The use of an instance size or an instance type depends on your cloud provider. If you do not want to define the instance size or type manually, you must add an annotation to define the use of an automatic instance size or type, based on the memory available.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
-
You have created a secret object and peer-pod
ConfigMap
unique to your cloud provider. -
You have created a
KataConfig
custom resource (CR).
Procedure
- From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
- In the workload page, click to create the workload.
-
In the YAML file for the workload, in the spec field where the container is listed, add
runtimeClassName: kata-remote
. In the YAML file for the workload, add an annotation to define whether to use a default instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.
For a specific instance size type, add the following annotation:
io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>
Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the
ConfigMap
for peer pods. Choose from one of those.For an automatic instance size or type, add the following annotations:
io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory>
Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.
Example for
Pod
objectapiVersion: v1 kind: Pod metadata: name: hello-openshift labels: app: hello-openshift annotations: io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1 spec: runtimeClassName: kata-remote containers: - name: hello-openshift image: quay.io/openshift/origin-hello-openshift ports: - containerPort: 8888 securityContext: privileged: false allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault
- 1
- This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.
- Click Save.
OpenShift Container Platform creates the workload and begins scheduling it.
3.3. Deploying OpenShift sandboxed containers workloads using peer pods with the CLI
You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig
custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata-remote
as the runtimeClassName
to the workload YAML file.
3.3.1. Installing the OpenShift sandboxed containers Operator using the CLI
You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI.
Prerequisites
- You have OpenShift Container Platform 4.15 installed on your cluster.
For installations on IBM Z or IBM® LinuxONE, you must have OpenShift Container Platform 4.14 or later installed.
ImportantDeploying OpenShift sandboxed containers workloads on IBM Z using peer pods is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. You have subscribed to the OpenShift sandboxed containers catalog.
NoteSubscribing to the OpenShift sandboxed containers catalog provides
openshift-sandboxed-containers-operator
namespace access to the OpenShift sandboxed containers Operator.
Procedure
Create the
Namespace
object for the OpenShift sandboxed containers Operator.Create a
Namespace
object YAML file that contains the following manifest:apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
Create the
Namespace
object:$ oc create -f Namespace.yaml
Create the
OperatorGroup
object for the OpenShift sandboxed containers Operator.Create an
OperatorGroup
object YAML file that contains the following manifest:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
Create the
OperatorGroup
object:$ oc create -f OperatorGroup.yaml
Create the
Subscription
object to subscribe theNamespace
to the OpenShift sandboxed containers Operator.Create a
Subscription
object YAML file that contains the following manifest:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.5.3
Create the
Subscription
object:$ oc create -f Subscription.yaml
The OpenShift sandboxed containers Operator is now installed on your cluster.
All the object file names listed above are suggestions. You can create the object YAML files using other names.
Verification
Ensure that the Operator is correctly installed:
$ oc get csv -n openshift-sandboxed-containers-operator
Example output
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.5.3 1.5.2 Succeeded
Additional resources
3.3.2. Setting up peer pods for AWS using the CLI
To set up peer pods for use on AWS, you must create a secret object, an AWS image VM (AMI), and a peer-pod ConfigMap
.
After setting up peer pods for AWS, you must still create the KataConfig
custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
3.3.2.1. Creating a secret object for AWS using the CLI
Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.
When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:
-
AWS_ACCESS_KEY_ID
-
AWS_SECRET_ACCESS_KEY
You can generate these values in the AWS console.
Procedure
Collect the necessary parameter values for the secret object. Make sure to write down each value.
Retrieve the instance ID:
$ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
This value is not needed for the secret object itself, but is used to retrieve other values for the secret object.
Retrieve the AWS region:
$ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""
Retrieve the AWS subnet ID:
$ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""
Retrieve the AWS VPC ID:
$ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""
Retrieve the AWS security group IDs:
$ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""
Create a YAML file with the following manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<enter value>" 1 AWS_SECRET_ACCESS_KEY: "<enter value>" 2 AWS_REGION: "<enter value>" 3 AWS_SUBNET_ID: "<enter value>" 4 AWS_VPC_ID: "<enter value>" 5 AWS_SG_IDS: "<enter value>" 6
- 1
- Enter the
AWS_ACCESS_KEY_ID
value you prepared before you began. - 2
- Enter the
AWS_SECRET_ACCESS_KEY
value you prepared before you began. - 3
- Enter the
AWS_REGION
value you retrieved. - 4
- Enter the
AWS_SUBNET_ID
value you retrieved. - 5
- Enter the
AWS_VPC_ID
value you retrieved. - 6
- Enter the
AWS_SG_IDS
value you retrieved.
Apply the secret object:
$ oc apply -f peer-pods-secret.yaml
The secret object is applied.
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the secret, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.3.2.2. Creating a peer-pod ConfigMap for AWS using the CLI
When creating a ConfigMap
for AWS, you must set the AMI ID. You can retrieve this value before you create the ConfigMap
.
Procedure
Create a YAML file with the following manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m"
Apply the
ConfigMap
:$ oc apply -f peer-pods-cm.yaml
The ConfigMap
is applied.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
Once you create the KataConfig
CR, you can run OpenShift sandboxed containers using peer pods on AWS.
3.3.3. Setting up peer pods for Azure using the CLI
To set up peer pods for use on Microsoft Azure, you must create a secret object, an Azure image VM, a peer-pod ConfigMap
, and an SSH key secret object.
After setting up peer pods for Azure, you must still create the KataConfig
custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
- You must have the Azure CLI tool installed and configured.
3.3.3.1. Creating a secret object for Azure using the CLI
Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.
When creating a secret object for Azure, you must set specific environment values. You can retrieve these values before creating the secret object. In addition, you must create a role-based access control (RBAC) file. This file generates the following values:
-
AZURE_CLIENT_ID
-
AZURE_CLIENT_SECRET
-
AZURE_TENANT_ID
Procedure
Retrieve the Azure subscription ID:
$ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
Generate the RBAC content. This generates the client ID, client secret, and the tenant ID:
$ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query "{ client_id: appId, client_secret: password, tenant_id: tenant }
The following output appears:
{ "client_id": `AZURE_CLIENT_ID`, "client_secret": `AZURE_CLIENT_SECRET`, "tenant_id": `AZURE_TENANT_ID` }
- Save the values in the RBAC output to use in the secret object.
Collect the additional parameter values for the secret object. Make sure to write down each value.
Retrieve the resource group:
$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
Retrieve the Azure region:
$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
Create a YAML file with the following manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AZURE_CLIENT_ID: "<enter value>" 1 AZURE_CLIENT_SECRET: "<enter value>" 2 AZURE_TENANT_ID: "<enter value>" 3 AZURE_SUBSCRIPTION_ID: "<enter value>" 4 AZURE_REGION: "<enter value>" 5 AZURE_RESOURCE_GROUP: "<enter value>" 6
- 1
- Enter the
AZURE_CLIENT_ID
value you generated in the RBAC file. - 2
- Enter the
AZURE_CLIENT_SECRET
value you generated in the RBAC file. - 3
- Enter the
AZURE_TENANT_ID
value you generated in the RBAC file. - 4
- Enter the
AZURE_SUBSCRIPTION_ID
value you retrieved. - 5
- Enter the
AZURE_REGION
value you retrieved. - 6
- Enter the
AZURE_RESOURCE_GROUP
value you retrieved.
Apply the secret object:
$ oc apply -f peer-pods-secret.yaml
The secret object is applied.
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the secret, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.3.3.2. Creating a peer-pod ConfigMap for Azure using the CLI
When creating a ConfigMap
for Azure, you must set specific configuration values. You can retrieve these values before you create the ConfigMap
.
Procedure
Collect the configuration values for the Azure peer-pod
ConfigMap
. Make sure to write down each value.Retrieve the Azure VNet name:
$ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
This value is not needed for the
ConfigMap
, but is used to retrieve the Azure subnet ID.Retrieve the Azure subnet ID:
$ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
Retrieve the Azure network security group (NSG) ID:
$ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
Create a YAML file with the following manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "azure" VXLAN_PORT: "9000" AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1 AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2 AZURE_SUBNET_ID: "<enter value>" 3 AZURE_NSG_ID: "<enter value>" 4 PROXY_TIMEOUT: "5m" DISABLECVM: "true"
- 1
- Defines the default instance size that is used when an instance is not defined in the workload.
- 2
- List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
- 3
- Enter the
AZURE_SUBNET_ID
value that you retrieved. - 4
- Enter the
AZURE_NSG_ID
value that you retrieved.
Apply the
ConfigMap
:$ oc apply -f peer-pods-cm.yaml
The ConfigMap
is deployed.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Keep in mind that restarting the daemonset recreates the peer pods, it does not update the existing pods.
3.3.3.3. Creating an SSH key secret object for Azure using the CLI
You must generate an SSH key and create an SSH key secret object to use peer pods with Azure.
Procedure
Generate the SSH key:
$ ssh-keygen -f ./id_rsa -N ""
Create the SSH secret object:
$ oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa
The SSH key is created and the SSH key secret object is created. Once you create the KataConfig
CR, you can run OpenShift sandboxed containers using peer pods on Azure.
3.3.4. Setting up peer pods for IBM Z using the CLI
To set up peer pods for use on an OpenShift Container Platform cluster running on IBM Z, you must create a secret object, a RHEL KVM image VM, and a peer-pod ConfigMap
, which hold the credentials that are required for the communication between libvirt and the KVM host.
After setting up peer pods for IBM Z, you must create the KataConfig
custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.
Prerequisites
- You have installed OpenShift Container Platform 4.14 or later on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
- You have libvirt installed on your KVM host and you have administrator privileges.
3.3.4.1. Setting up libvirt on the KVM host
You must set up libvirt on the KVM host. Peer pods on IBM Z use the libvirt provider of the cloud API Adaptor to create and manage virtual machines.
Procedure
Log in to IBM Z KVM host and set the shell variables to be used by libvirt for storage management.
Set the name of the libvirt pool by running the following command:
$ export LIBVIRT_POOL=<name_of_libvirt_pool_to_create>
Set the name of the libvirt pool by running the following command:
$ export LIBVIRT_VOL_NAME=<name_of_libvirt_volume_to_create>
Set the path of the default storage pool location, by running the following command:
$ export LIBVIRT_POOL_DIRECTORY=<name_of_target_directory> 1
- 1
- To ensure libvirt has read and write access permissions, use a subdirectory of libvirt’s storage directory. The default is
/var/lib/libvirt/images/
.
Create a libvirt pool by running the following command:
$ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"
Start the libvirt pool by running the following command:
$ virsh pool-start $LIBVIRT_POOL
Create a libvirt volume for the pool by running the following command:
$ virsh -c qemu:///system \ vol-create-as --pool $LIBVIRT_POOL \ --name $LIBVIRT_VOL_NAME \ --capacity 20G \ --allocation 2G \ --prealloc-metadata \ --format qcow2
3.3.4.2. Creating a peer-pod VM image for IBM Z
To run OpenShift sandboxed containers using peer pods on IBM Z, you must first create a KVM guest from which the libvirt provider starts the peer-pod VM.
Once the image is created, upload the image to the volume that you created in the previous step.
Prerequisites
- IBM z15 or later, or IBM® LinuxONE III or later.
- At least one LPAR running on RHEL 9 or later with KVM.
Procedure
Set the RHEL
ORG_ID
andACTIVATION_KEY
shell variables for your system.When using a subscribed RHEL system, set the shell environment variables to the files holding the organization ID and the activation key for Red Hat Subscription Management (RHSM) by running the following commands:
$ export ORG_ID=$(cat ~/.rh_subscription/orgid) $ export ACTIVATION_KEY=$(cat ~/.rh_subscription/activation_key)
When using an unsubscribed RHEL system, set the appropriate subscription values by running the following commands:
$ export ORG_ID=<RHEL_ORGID_VALUE> $ export ACTIVATION_KEY=<RHEL_ACVTIVATION_KEY>
Log in to your IBM Z system and perform the following steps:
-
Download an
s390x
RHEL KVM guest image from the Red Hat Customer Portal to your libvirt storage directory to grant libvirt correct access. The default directory is/var/lib/libvirt/images
. The image is used to generate the peer pod VM image, which will include the relevant binaries. Set the
IMAGE_URL
shell environment variable for the downloaded image by running the following command:$ export IMAGE_URL=<location_of_downloaded_KVM_guest_image> 1
- 1
- Enter the path of the KVM guest image that you downloaded in the previous step.
Register the guest KVM image by running the following command:
$ export REGISTER_CMD="subscription-manager register --org=${ORG_ID} \ --activationkey=${ACTIVATION_KEY}"
Customize the guest KVM image by running the following command:
$ virt-customize -v -x -a ${IMAGE_URL} --run-command "${REGISTER_CMD}"
Set the checksum of the image by running the following command:
$ export IMAGE_CHECKSUM=$(sha256sum ${IMAGE_URL} | awk '{ print $1 }')
-
Download an
3.3.4.2.1. Building the peer-pod VM QCOW2 image
To run OpenShift sandboxed containers using peer pods on IBM Z, you must build a peer-pod VM QCOW2 image.
Procedure
Clone the cloud-api-adaptor repository onto your build workstation by running the following command:
$ git clone --single-branch https://github.com/confidential-containers/cloud-api-adaptor.git
Change into the
podvm
directory by running the following command:$ cd cloud-api-adaptor && git checkout 8577093
Create a builder image from which the final QCOW2 image is generated.
When using a subscribed RHEL system, run the following command:
$ podman build -t podvm_builder_rhel_s390x \ --build-arg ARCH="s390x" \ --build-arg GO_VERSION="1.21.3" \ --build-arg PROTOC_VERSION="25.1" \ --build-arg PACKER_VERSION="v1.9.4" \ --build-arg RUST_VERSION="1.72.0" \ --build-arg YQ_VERSION="v4.35.1" \ --build-arg YQ_CHECKSUM="sha256:4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \ -f podvm/Dockerfile.podvm_builder.rhel .
When using an unsubscribed RHEL system, run the following command:
$ podman build -t podvm_builder_rhel_s390x \ --build-arg ORG_ID=$ORG_ID \ --build-arg ACTIVATION_KEY=$ACTIVATION_KEY \ --build-arg ARCH="s390x" \ --build-arg GO_VERSION="1.21.3" \ --build-arg PROTOC_VERSION="25.1" \ --build-arg PACKER_VERSION="v1.9.4" \ --build-arg RUST_VERSION="1.72.0" \ --build-arg YQ_VERSION="v4.35.1" \ --build-arg YQ_CHECKSUM="sha256:4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \ -f podvm/Dockerfile.podvm_builder.rhel .
Generate an intermediate image package with the required binaries for running peer pods by running the following command:
$ podman build -t podvm_binaries_rhel_s390x \ --build-arg BUILDER_IMG="podvm_builder_rhel_s390x:latest" \ --build-arg ARCH=s390x \ -f podvm/Dockerfile.podvm_binaries.rhel .
NoteThis process is expected to take significant time.
Extract the binaries and build the peer-pod QCOW2 image by running the following command:
$ podman build -t podvm_rhel_s390x \ --build-arg ARCH=s390x \ --build-arg CLOUD_PROVIDER=libvirt \ --build-arg BUILDER_IMG="localhost/podvm_builder_rhel_s390x:latest" \ --build-arg BINARIES_IMG="localhost/podvm_binaries_rhel_s390x:latest" \ -v ${IMAGE_URL}:/tmp/rhel.qcow2:Z \ --build-arg IMAGE_URL="/tmp/rhel.qcow2" \ --build-arg IMAGE_CHECKSUM=${IMAGE_CHECKSUM} \ -f podvm/Dockerfile.podvm.rhel .
Extract the peer pod QCOW2 image to a directory of your choice by running the following commands:
$ export IMAGE_OUTPUT_DIR=<image_output_directory> 1 $ mkdir -p $IMAGE_OUTPUT_DIR $ podman save podvm_rhel_s390x | tar -xO --no-wildcards-match-slash '*.tar' | tar -x -C ${IMAGE_OUTPUT_DIR}
- 1
- Enter the
image_output_directory
where to extract the final QCOW image.
Upload the peer pod QCOW2 image to your libvirt volume:
$ virsh -c qemu:///system vol-upload \ --vol $LIBVIRT_VOL_NAME \ $IMAGE_OUTPUT_DIR/podvm-*.qcow2 \ --pool $LIBVIRT_POOL --sparse
3.3.4.3. Creating a RHEL secret for peer-pod credentials
When creating a secret object for IBM Z, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:
-
LIBVIRT_POOL
-
LIBVIRT_VOL_NAME
-
LIBVIRT_URI
LIBVIRT_URI
is the default gateway IP address of your libvirt network. Check your libvirt network setup to obtain this value.
If your libvirt installation is using the default bridge virtual network, you can obtain the LIBVIRT_URI
by running the following commands:
$ virtint=$(bridge_line=$(virsh net-info default | grep Bridge); echo "${bridge_line//Bridge:/}" | tr -d [:blank:]) $ LIBVIRT_URI=$( ip -4 addr show $virtint | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
Procedure
Create a YAML file
peer-pods-secret.yaml
with the following manifest:apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: CLOUD_PROVIDER: "libvirt" 1 LIBVIRT_URI: "<libvirt_gateway_uri>" 2 LIBVIRT_POOL: "<libvirt_pool>" 3 LIBVIRT_VOL_NAME: "<libvirt_volume>" 4
Create the secret object:
$ oc apply -f peer-pods-secret.yaml
The secret object is applied.
3.3.4.4. Creating a peer-pod ConfigMap for IBM Z using the CLI
When creating a ConfigMap
for IBM Z, you must use the libvirt provider.
Procedure
Create a YAML file
peer-pods-cm.yaml
with the following manifest:apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "libvirt" PROXY_TIMEOUT: "15m"
Apply the
ConfigMap
:$ oc apply -f peer-pods-cm.yaml
The ConfigMap
is applied.
3.3.4.5. Creating an SSH key secret object for IBM Z using the CLI
You must generate an SSH key pair and create an SSH key secret object to use peer pods with IBM Z.
Procedure
Generate the SSH key:
$ ssh-keygen -f ./id_rsa -N ""
Copy the SSH public key to your KVM host:
$ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_ADDRESS> 1
- 1
- Enter the IP address of your KVM host.
Create the secret object:
$ oc create secret generic ssh-key-secret \ -n openshift-sandboxed-containers-operator \ --from-file=id_rsa.pub=./id_rsa.pub \ --from-file=id_rsa=./id_rsa
Delete the SSH keys:
$ shred –remove id_rsa.pub id_rsa
The secret object is created. Once you create the KataConfig
CR, you can run OpenShift sandboxed containers using peer pods on IBM Z.
3.3.5. Creating the KataConfig custom resource using the CLI
You must create one KataConfig
custom resource (CR) to install kata-remote
as a RuntimeClass
on your nodes. Creating the KataConfig
CR triggers the OpenShift sandboxed containers Operator to do the following:
-
Install the needed RHCOS extensions, such as QEMU and
kata-containers
, on your RHCOS node. - Ensure that the CRI-O runtime is configured with the correct runtime handlers.
-
Create a
RuntimeClass
CR namedkata-remote
with a default configuration. This enables users to configure workloads to usekata-remote
as the runtime by referencing the CR in theRuntimeClassName
field. This CR also specifies the resource overhead for the runtime.
Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote
as a RuntimeClass
only on specific nodes, you can add labels to those nodes, and then define the label in the KataConfig
CR when you create it.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Procedure
Create a YAML file with the following manifest:
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: enablePeerPods: true logLevel: info
(Optional) If you want to install
kata-remote
as aRuntimeClass
only on selected nodes, create a YAML file that includes the label in the manifest:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: enablePeerPods: true logLevel: info kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
- 1
- Labels in
kataConfigPoolSelector
only support single values;nodeSelector
syntax is not supported.
Create the
KataConfig
resource:$ oc create -f cluster-kataconfig.yaml
The new KataConfig
CR is created and begins to install kata-remote
as a RuntimeClass
on the worker nodes. Wait for the kata-remote
installation to complete and the worker nodes to reboot before continuing to the next step.
When you run the CR, the VM image is created. The image creation is done through the cloud provider and it may use additional resources.
OpenShift sandboxed containers installs kata-remote
only as a secondary, optional runtime on the cluster and not as the primary runtime.
Verification
Monitor the installation progress:
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
When all workers under kataNodes are listed as
installed
, and the conditionInProgress
isFalse
without specifying a reason, this indicates thatkata
is installed on the cluster. For more information, see "Installation and uninstall transitions".
Additional resources
3.3.6. Deploying a workload with peer pods in a sandboxed container using the CLI
OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.
To deploy a pod-templated workload using peer pods in a sandboxed container, you must add kata-remote
as the runtimeClassName
to the workload YAML file.
You must also define whether the workload should be deployed using a default instance size or type which you previously defined in the ConfigMap
by adding an annotation to the YAML file. The use of an instance size or an instance type depends on your cloud provider. If you do not want to define the instance size or type manually, you must add an annotation to define the use of an automatic instance size or type, based on the memory available.
Prerequisites
- You have installed OpenShift Container Platform 4.15 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
-
You have created a secret object and peer-pod
ConfigMap
unique to your cloud provider. -
You have created a
KataConfig
custom resource (CR).
Procedure
Add
runtimeClassName: kata-remote
to any pod-templated object:-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
-
Add an annotation to the pod-templated object, defining whether to use a specific instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.
For a specific instance size or type, add the following annotation:
io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>
Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the
ConfigMap
for peer pods. Choose from one of those.For an automatic instance size or type, add the following annotations:
io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory>
Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.
Example for
Pod
objectapiVersion: v1 kind: Pod metadata: name: hello-openshift labels: app: hello-openshift annotations: io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1 spec: runtimeClassName: kata-remote containers: - name: hello-openshift image: quay.io/openshift/origin-hello-openshift ports: - containerPort: 8888 securityContext: privileged: false allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault
- 1
- This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.
OpenShift Container Platform creates the workload and begins scheduling it.
Verification
-
Inspect the
runtimeClassName
field on a pod-templated object. If theruntimeClassName
iskata-remote
, then the workload is running on OpenShift sandboxed containers, using peer pods.
3.4. Additional resources
- The OpenShift sandboxed containers Operator is supported in a restricted network environment. For more information, see Using Operator Lifecycle Manager on restricted networks.
- When using a disconnected cluster on a restricted network, you must configure proxy support in Operator Lifecycle Manager to access the OperatorHub. Using a proxy allows the cluster to fetch the OpenShift sandboxed containers Operator.