Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 3. Deploying workloads on public cloud
You can deploy OpenShift sandboxed containers workloads on AWS Cloud Computing Services and Microsoft Azure Cloud Computing Services.
Cluster requirements
- You have installed Red Hat OpenShift Container Platform 4.13 or later.
- Your cluster has at least one worker node.
3.1. Deploying workloads on AWS
You can deploy OpenShift sandboxed containers workloads on AWS Cloud Computing Services by using the OpenShift Container Platform web console or the command line interface (CLI).
Deployment workflow
- Enable ports.
- Create a secret for AWS.
- Create a config map for AWS.
-
Create a
KataConfig
custom resource. - Optional: Modify the peer pod VM limit per node.
-
Configure your workload objects to use the
kata-remote
runtime class.
3.1.1. Preparing your environment
Perform the following steps to prepare your environment:
- Ensure that your cluster has sufficient resources.
- Install the OpenShift sandboxed containers Operator.
- Enable ports 15150 and 9000 to allow internal communication with peer pods.
3.1.1.1. Resource requirements
Peer pod virtual machines (VMs) require resources in two locations:
-
The worker node. The worker node stores metadata, Kata shim resources (
containerd-shim-kata-v2
), remote-hypervisor resources (cloud-api-adaptor
), and the tunnel setup between the worker nodes and the peer pod VM. - The cloud instance. This is the actual peer pod VM running in the cloud.
The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote
) definition used for creating peer pods.
The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit
attribute in the peerpodConfig
custom resource (CR).
The peerpodConfig
CR, named peerpodconfig-openshift
, is created when you create the kataConfig
CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator
namespace.
The following peerpodConfig
CR example displays the default spec
values:
apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
name: peerpodconfig-openshift
namespace: openshift-sandboxed-containers-operator
spec:
cloudSecretName: peer-pods-secret
configMapName: peer-pods-cm
limit: "10" 1
nodeSelector:
node-role.kubernetes.io/kata-oc: ""
- 1
- The default limit is 10 VMs per node.
The extended resource is named kata.peerpods.io/vm
, and enables the Kubernetes scheduler to handle capacity tracking and accounting.
You can edit the limit per node based on the requirements for your environment. See "Modifying the VM limit per node in peer pods" for more information.
A mutating webhook adds the extended resource kata.peerpods.io/vm
to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available.
The mutating webhook modifies a Kubernetes pod as follows:
-
The mutating webhook checks the pod for the expected
RuntimeClassName
value, specified in theTARGET_RUNTIME_CLASS
environment variable. If the value in the pod specification does not match the value in theTARGET_RUNTIME_CLASS
, the webhook exits without modifying the pod. If the
RuntimeClassName
values match, the webhook makes the following changes to the pod spec:-
The webhook removes every resource specification from the
resources
field of all containers and init containers in the pod. -
The webhook adds the extended resource (
kata.peerpods.io/vm
) to the spec by modifying the resources field of the first container in the pod. The extended resourcekata.peerpods.io/vm
is used by the Kubernetes scheduler for accounting purposes.
-
The webhook removes every resource specification from the
The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.
As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces.
3.1.1.2. Enabling ports for AWS
You must enable ports 15150 and 9000 to allow internal communication with peer pods running on AWS.
Prerequisites
- You have installed the OpenShift sandboxed containers Operator.
- You have installed the AWS command line tool.
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
Log in to your OpenShift Container Platform cluster and retrieve the instance ID:
$ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
Retrieve the AWS region:
$ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
Retrieve the security group IDs and store them in an array:
$ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region $AWS_REGION))
For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:
$ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION done
The ports are now enabled.
3.1.1.3. Installing the OpenShift sandboxed containers Operator
You can install the OpenShift sandboxed containers Operator by using the OpenShift Container Platform web console or command line interface (CLI).
3.1.1.3.1. Installing the Operator by using the web console
You can install the OpenShift sandboxed containers Operator by using the Red Hat OpenShift Container Platform web console.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
In the OpenShift Container Platform web console, navigate to Operators
OperatorHub. -
In the Filter by keyword field, type
OpenShift sandboxed containers
. - Select the OpenShift sandboxed containers Operator tile and click Install.
- On the Install Operator page, select stable from the list of available Update Channel options.
Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory
openshift-sandboxed-containers-operator
namespace. If this namespace does not yet exist, it is automatically created.NoteAttempting to install the OpenShift sandboxed containers Operator in a namespace other than
openshift-sandboxed-containers-operator
causes the installation to fail.- Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
- Click Install.
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
-
Navigate to Operators
Installed Operators. - Verify that the OpenShift sandboxed containers Operator is displayed.
Additional resources
- Using Operator Lifecycle Manager on restricted networks.
- Configuring proxy support in Operator Lifecycle Manager for disconnected environments.
3.1.1.3.2. Installing the Operator by using the CLI
You can install the OpenShift sandboxed containers Operator by using the CLI.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
Create a
Namespace.yaml
manifest file:apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
Create the namespace by running the following command:
$ oc create -f Namespace.yaml
Create an
OperatorGroup.yaml
manifest file:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
Create the operator group by running the following command:
$ oc create -f OperatorGroup.yaml
Create a
Subscription.yaml
manifest file:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.6.0
Create the subscription by running the following command:
$ oc create -f Subscription.yaml
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
Ensure that the Operator is correctly installed by running the following command:
$ oc get csv -n openshift-sandboxed-containers-operator
Example output
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.6.0 1.5.3 Succeeded
3.1.1.3.3. Additional resources
- Using Operator Lifecycle Manager on restricted networks
- Configuring proxy support in Operator Lifecycle Manager for disconnected environments
3.1.2. Deploying workloads by using the web console
You can deploy OpenShift sandboxed containers workloads by using the web console.
3.1.2.1. Creating a secret
You must create a Secret
object on your OpenShift Container Platform cluster. The secret stores cloud provider credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.
Prerequisites
-
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
You can generate these values in the AWS console.
Procedure
-
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Click the OpenShift sandboxed containers Operator tile.
- Click the Import icon (+) on the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2
- Click Save to apply the changes.
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
DaemonSet to apply the changes.
After you update the secret, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting a daemon set recreates peer pods. It does not update existing pods.
Verification
-
Navigate to Workloads
Secrets to view the secret.
3.1.2.2. Creating a config map
You must create a config map on your OpenShift Container Platform cluster for your cloud provider.
You must set the Amazon Machine Image (AMI) ID. You can retrieve this value before you create the config map.
Procedure
Obtain the following values from your AWS instance:
Retrieve and record the instance ID:
$ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
This is used to retrieve other values for the secret object.
Retrieve and record the AWS region:
$ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""
Retrieve and record the AWS subnet ID:
$ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""
Retrieve and record the AWS VPC ID:
$ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""
Retrieve and record the AWS security group IDs:
$ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region ${AWS_REGION} --output text) && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""
-
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" DISABLECVM: "true" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7
- 1
- Defines the default instance type that is used when a type is not defined in the workload.
- 2
- Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.
- 3
- Optional: By default, this value is populated when you run the
KataConfig
CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID. - 4
- Specify the
AWS_REGION
value you retrieved. - 5
- Specify the
AWS_SUBNET_ID
value you retrieved. - 6
- Specify the
AWS_VPC_ID
value you retrieved. - 7
- Specify the
AWS_SG_IDS
value you retrieved.
Click Save to apply the changes.
A config map is created for your cloud provider.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting the daemonset recreates the peer pods. It does not update the existing pods.
Verification
-
Navigate to Workloads
ConfigMaps to view the new config map.
3.1.2.3. Creating a KataConfig custom resource
You must create a KataConfig
custom resource (CR) to install kata-remote
as a RuntimeClass
on your worker nodes.
The kata-remote
runtime class is installed on all worker nodes by default. If you want to install kata-remote
only on specific nodes, you can add labels to those nodes and then define the label in the KataConfig
CR.
OpenShift sandboxed containers installs kata-remote
as a secondary, optional runtime on the cluster and not as the primary runtime.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors might increase the reboot time:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator.
- On the KataConfig tab, click Create KataConfig.
Enter the following details:
-
Name: Optional: The default name is
example-kataconfig
. -
Labels: Optional: Enter any relevant, identifying attributes to the
KataConfig
resource. Each label represents a key-value pair. - enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.
kataConfigPoolSelector. Optional: To install
kata-remote
on selected nodes, add a match expression for the labels on the selected nodes:- Expand the kataConfigPoolSelector area.
- In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
- Click Add matchExpressions.
- In the Key field, enter the label key the selector applies to.
-
In the Operator field, enter the key’s relationship to the label values. Valid operators are
In
,NotIn
,Exists
, andDoesNotExist
. - Expand the Values area and then click Add value.
-
In the Value field, enter
true
orfalse
for key label value.
-
logLevel: Define the level of log data retrieved for nodes with the
kata-remote
runtime class.
-
Name: Optional: The default name is
Click Create. The
KataConfig
CR is created and installs thekata-remote
runtime class on the worker nodes.Wait for the
kata-remote
installation to complete and the worker nodes to reboot before verifying the installation.
Verification
-
On the KataConfig tab, click the
KataConfig
CR to view its details. Click the YAML tab to view the
status
stanza.The
status
stanza contains theconditions
andkataNodes
keys. The value ofstatus.kataNodes
is an array of nodes, each of which lists nodes in a particular state ofkata-remote
installation. A message appears each time there is an update.Click Reload to refresh the YAML.
When all workers in the
status.kataNodes
array display the valuesinstalled
andconditions.InProgress: False
with no specified reason, thekata-remote
is installed on the cluster.
See KataConfig status messages for details.
3.1.2.3.1. Optional: Verifying the pod VM image
After kata-remote
is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.
Procedure
-
Navigate to Workloads
ConfigMaps. - Click the provider config map to view its details.
- Click the YAML tab.
Check the
status
stanza of the YAML file.If the
PODVM_AMI_ID
parameter is populated, the pod VM image was created successfully.
Troubleshooting
Retrieve the events log by running the following command:
$ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
Retrieve the job log by running the following command:
$ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.
3.1.2.4. Optional: Modifying the number of peer pod VMs per node
You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig
custom resource (CR).
Procedure
Check the current limit by running the following command:
$ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}'
Modify the
limit
attribute of thepeerpodConfig
CR by running the following command:$ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1
- 1
- Replace <value> with the limit you want to define.
3.1.2.5. Configuring workload objects
You deploy an OpenShift sandboxed containers workload by configuring kata-remote
as the runtime class for the following pod-templated objects:
-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
Do not deploy workloads in the openshift-sandboxed-containers-operator
namespace. Create a dedicated namespace for these resources.
You can define whether the workload should be deployed using the default instance type, which you defined in the config map, by adding an annotation to the YAML file.
If you do not want to define the instance type manually, you can add an annotation to use an automatic instance type, based on the memory available.
Prerequisites
- You have created a secret object for your provider.
- You have created a config map for your provider.
-
You have created a
KataConfig
custom resource (CR).
Procedure
-
In the OpenShift Container Platform web console, navigate to Workloads
workload type, for example, Pods. - On the workload type page, click an object to view its details.
- Click the YAML tab.
Add
spec.runtimeClassName: kata-remote
to the manifest of each pod-templated workload object as in the following example:apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ...
Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:
To use a manually defined instance type, add the following annotation:
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: t3.medium 1 # ...
- 1
- Specify the instance type that you defined in the config map.
To use an automatic instance type, add the following annotations:
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ...
Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.
Click Save to apply the changes.
OpenShift Container Platform creates the workload object and begins scheduling it.
Verification
-
Inspect the
spec.runtimeClassName
field of a pod-templated object. If the value iskata-remote
, then the workload is running on OpenShift sandboxed containers, using peer pods.
3.1.3. Deploying workloads by using the command line
You can deploy OpenShift sandboxed containers workloads by using the command line.
3.1.3.1. Creating a secret
You must create a Secret
object on your OpenShift Container Platform cluster. The secret stores cloud provider credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.
Prerequisites
-
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
You can generate these values in the AWS console.
Procedure
Create a
peer-pods-secret.yaml
manifest file according to the following example:apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AWS_ACCESS_KEY_ID: "<aws_access_key>" 1 AWS_SECRET_ACCESS_KEY: "<aws_secret_access_key>" 2
Create the
secret
object by applying the manifest:$ oc apply -f peer-pods-secret.yaml
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
DaemonSet to apply the changes.
After you update the secret, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting a daemon set recreates peer pods. It does not update existing pods.
3.1.3.2. Creating a config map
You must create a config map on your OpenShift Container Platform cluster for your cloud provider.
You must set the Amazon Machine Image (AMI) ID. You can retrieve this value before you create the config map.
Procedure
Obtain the following values from your AWS instance:
Retrieve and record the instance ID:
$ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
This is used to retrieve other values for the secret object.
Retrieve and record the AWS region:
$ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""
Retrieve and record the AWS subnet ID:
$ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""
Retrieve and record the AWS VPC ID:
$ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""
Retrieve and record the AWS security group IDs:
$ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region ${AWS_REGION} --output text) && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""
Create a
peer-pods-cm.yaml
manifest according to the following example:apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "aws" VXLAN_PORT: "9000" PODVM_INSTANCE_TYPE: "t3.medium" 1 PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2 PROXY_TIMEOUT: "5m" DISABLECVM: "true" PODVM_AMI_ID: "<podvm_ami_id>" 3 AWS_REGION: "<aws_region>" 4 AWS_SUBNET_ID: "<aws_subnet_id>" 5 AWS_VPC_ID: "<aws_vpc_id>" 6 AWS_SG_IDS: "<aws_sg_ids>" 7
- 1
- Defines the default instance type that is used when a type is not defined in the workload.
- 2
- Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.
- 3
- Optional: By default, this value is populated when you run the
KataConfig
CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID. - 4
- Specify the
AWS_REGION
value you retrieved. - 5
- Specify the
AWS_SUBNET_ID
value you retrieved. - 6
- Specify the
AWS_VPC_ID
value you retrieved. - 7
- Specify the
AWS_SG_IDS
value you retrieved.
Apply the manifest to create a config map:
$ oc apply -f peer-pods-cm.yaml
A config map is created for your cloud provider.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting the daemonset recreates the peer pods. It does not update the existing pods.
3.1.3.3. Creating a KataConfig custom resource
You must create a KataConfig
custom resource (CR) to install kata-remote
as a runtime class on your worker nodes.
Creating the KataConfig
CR triggers the OpenShift sandboxed containers Operator to do the following:
-
Create a
RuntimeClass
CR namedkata-remote
with a default configuration. This enables users to configure workloads to usekata-remote
as the runtime by referencing the CR in theRuntimeClassName
field. This CR also specifies the resource overhead for the runtime.
OpenShift sandboxed containers installs kata-remote
as a secondary, optional runtime on the cluster and not as the primary runtime.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
Create a
cluster-kataconfig.yaml
manifest file according to the following example:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: enablePeerPods: true logLevel: info
Optional: To install
kata-remote
on selected nodes, specify the node labels according to the following example:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1 # ...
- 1
- Specify the labels of the selected nodes.
Create the
KataConfig
CR:$ oc create -f cluster-kataconfig.yaml
The new
KataConfig
CR is created and installskata-remote
as a runtime class on the worker nodes.Wait for the
kata-remote
installation to complete and the worker nodes to reboot before verifying the installation.
Verification
Monitor the installation progress by running the following command:
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
When the status of all workers under
kataNodes
isinstalled
and the conditionInProgress
isFalse
without specifying a reason, thekata-remote
is installed on the cluster.
See KataConfig status messages for details.
3.1.3.3.1. Optional: Verifying the pod VM image
After kata-remote
is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.
Procedure
Obtain the config map you created for the peer pods:
$ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml
Check the
status
stanza of the YAML file.If the
PODVM_AMI_ID
parameter is populated, the pod VM image was created successfully.
Troubleshooting
Retrieve the events log by running the following command:
$ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
Retrieve the job log by running the following command:
$ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.
3.1.3.4. Optional: Modifying the number of peer pod VMs per node
You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig
custom resource (CR).
Procedure
Check the current limit by running the following command:
$ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}'
Modify the
limit
attribute of thepeerpodConfig
CR by running the following command:$ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1
- 1
- Replace <value> with the limit you want to define.
3.1.3.5. Configuring workload objects
You deploy an OpenShift sandboxed containers workload by configuring kata-remote
as the runtime class for the following pod-templated objects:
-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
Do not deploy workloads in the openshift-sandboxed-containers-operator
namespace. Create a dedicated namespace for these resources.
You can define whether the workload should be deployed using the default instance type, which you defined in the config map, by adding an annotation to the YAML file.
If you do not want to define the instance type manually, you can add an annotation to use an automatic instance type, based on the memory available.
Prerequisites
- You have created a secret object for your provider.
- You have created a config map for your provider.
-
You have created a
KataConfig
custom resource (CR).
Procedure
Add
spec.runtimeClassName: kata-remote
to the manifest of each pod-templated workload object as in the following example:apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ...
Add an annotation to the pod-templated object to use a manually defined instance type or an automatic instance type:
To use a manually defined instance type, add the following annotation:
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: t3.medium 1 # ...
- 1
- Specify the instance type that you defined in the config map.
To use an automatic instance type, add the following annotations:
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ...
Define the amount of memory available for the workload to use. The workload will run on an automatic instance type based on the amount of memory available.
Apply the changes to the workload object by running the following command:
$ oc apply -f <object.yaml>
OpenShift Container Platform creates the workload object and begins scheduling it.
Verification
-
Inspect the
spec.runtimeClassName
field of a pod-templated object. If the value iskata-remote
, then the workload is running on OpenShift sandboxed containers, using peer pods.
3.2. Deploying workloads on Azure
You can deploy OpenShift sandboxed containers workloads on Microsoft Azure Cloud Computing Services by using the OpenShift Container Platform web console or the command line interface (CLI).
Deployment workflow
- Create a secret for your Azure access keys.
- Create a config map to define Azure instance sizes and other parameters.
- Create an SSH key secret.
-
Create a
KataConfig
custom resource. - Optional: Modify the peer pod VM limit per node.
-
Configure your workload objects to use the
kata-remote
runtime class.
3.2.1. Preparing your environment
Perform the following steps to prepare your environment:
- Ensure that your cluster has sufficient resources.
- Install the OpenShift sandboxed containers Operator.
3.2.1.1. Resource requirements
Peer pod virtual machines (VMs) require resources in two locations:
-
The worker node. The worker node stores metadata, Kata shim resources (
containerd-shim-kata-v2
), remote-hypervisor resources (cloud-api-adaptor
), and the tunnel setup between the worker nodes and the peer pod VM. - The cloud instance. This is the actual peer pod VM running in the cloud.
The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote
) definition used for creating peer pods.
The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit
attribute in the peerpodConfig
custom resource (CR).
The peerpodConfig
CR, named peerpodconfig-openshift
, is created when you create the kataConfig
CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator
namespace.
The following peerpodConfig
CR example displays the default spec
values:
apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
name: peerpodconfig-openshift
namespace: openshift-sandboxed-containers-operator
spec:
cloudSecretName: peer-pods-secret
configMapName: peer-pods-cm
limit: "10" 1
nodeSelector:
node-role.kubernetes.io/kata-oc: ""
- 1
- The default limit is 10 VMs per node.
The extended resource is named kata.peerpods.io/vm
, and enables the Kubernetes scheduler to handle capacity tracking and accounting.
You can edit the limit per node based on the requirements for your environment. See "Modifying the VM limit per node in peer pods" for more information.
A mutating webhook adds the extended resource kata.peerpods.io/vm
to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available.
The mutating webhook modifies a Kubernetes pod as follows:
-
The mutating webhook checks the pod for the expected
RuntimeClassName
value, specified in theTARGET_RUNTIME_CLASS
environment variable. If the value in the pod specification does not match the value in theTARGET_RUNTIME_CLASS
, the webhook exits without modifying the pod. If the
RuntimeClassName
values match, the webhook makes the following changes to the pod spec:-
The webhook removes every resource specification from the
resources
field of all containers and init containers in the pod. -
The webhook adds the extended resource (
kata.peerpods.io/vm
) to the spec by modifying the resources field of the first container in the pod. The extended resourcekata.peerpods.io/vm
is used by the Kubernetes scheduler for accounting purposes.
-
The webhook removes every resource specification from the
The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.
As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces.
3.2.1.2. Installing the OpenShift sandboxed containers Operator
You can install the OpenShift sandboxed containers Operator by using the OpenShift Container Platform web console or command line interface (CLI).
3.2.1.2.1. Installing the Operator by using the web console
You can install the OpenShift sandboxed containers Operator by using the Red Hat OpenShift Container Platform web console.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
In the OpenShift Container Platform web console, navigate to Operators
OperatorHub. -
In the Filter by keyword field, type
OpenShift sandboxed containers
. - Select the OpenShift sandboxed containers Operator tile and click Install.
- On the Install Operator page, select stable from the list of available Update Channel options.
Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory
openshift-sandboxed-containers-operator
namespace. If this namespace does not yet exist, it is automatically created.NoteAttempting to install the OpenShift sandboxed containers Operator in a namespace other than
openshift-sandboxed-containers-operator
causes the installation to fail.- Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
- Click Install.
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
-
Navigate to Operators
Installed Operators. - Verify that the OpenShift sandboxed containers Operator is displayed.
Additional resources
- Using Operator Lifecycle Manager on restricted networks.
- Configuring proxy support in Operator Lifecycle Manager for disconnected environments.
3.2.1.2.2. Installing the Operator by using the CLI
You can install the OpenShift sandboxed containers Operator by using the CLI.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
Create a
Namespace.yaml
manifest file:apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
Create the namespace by running the following command:
$ oc create -f Namespace.yaml
Create an
OperatorGroup.yaml
manifest file:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
Create the operator group by running the following command:
$ oc create -f OperatorGroup.yaml
Create a
Subscription.yaml
manifest file:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: stable installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.6.0
Create the subscription by running the following command:
$ oc create -f Subscription.yaml
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
Ensure that the Operator is correctly installed by running the following command:
$ oc get csv -n openshift-sandboxed-containers-operator
Example output
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.6.0 1.5.3 Succeeded
3.2.1.2.3. Additional resources
- Using Operator Lifecycle Manager on restricted networks
- Configuring proxy support in Operator Lifecycle Manager for disconnected environments
3.2.2. Deploying workloads by using the web console
You can deploy OpenShift sandboxed containers workloads by using the web console.
3.2.2.1. Creating a secret
You must create a Secret
object on your OpenShift Container Platform cluster. The secret stores cloud provider credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.
Prerequisites
- You have installed and configured the Azure CLI tool.
Procedure
Retrieve the Azure subscription ID:
$ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
Generate the RBAC content. This generates the client ID, client secret, and the tenant ID:
$ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query "{ client_id: appId, client_secret: password, tenant_id: tenant }
Example output:
{ "client_id": `AZURE_CLIENT_ID`, "client_secret": `AZURE_CLIENT_SECRET`, "tenant_id": `AZURE_TENANT_ID` }
-
Record the RBAC output to use in the
secret
object. -
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Click the OpenShift sandboxed containers Operator tile.
- Click the Import icon (+) on the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AZURE_CLIENT_ID: "<azure_client_id>" 1 AZURE_CLIENT_SECRET: "<azure_client_secret>" 2 AZURE_TENANT_ID: "<azure_tenant_id>" 3 AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>" 4
- Click Save to apply the changes.
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
DaemonSet to apply the changes.
After you update the secret, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting a daemon set recreates peer pods. It does not update existing pods.
Verification
-
Navigate to Workloads
Secrets to view the secret.
3.2.2.2. Creating a config map
You must create a config map on your OpenShift Container Platform cluster for your cloud provider.
Procedure
Obtain the following values from your Azure instance:
Retrieve and record the Azure VNet name:
$ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
This value is used to retrieve the Azure subnet ID.
Retrieve and record the Azure subnet ID:
$ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
Retrieve and record the Azure network security group (NSG) ID:
$ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
Retrieve and record the Azure resource group:
$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
Retrieve and record the Azure region:
$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
-
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- Click the Import icon (+) in the top right corner.
In the Import YAML window, paste the following YAML manifest:
apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "azure" VXLAN_PORT: "9000" AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1 AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2 AZURE_SUBNET_ID: "<azure_subnet_id>" 3 AZURE_NSG_ID: "<azure_nsg_id>" 4 PROXY_TIMEOUT: "5m" DISABLECVM: "true" AZURE_IMAGE_ID: "<azure_image_id>" 5 AZURE_REGION: "<azure_region>" 6 AZURE_RESOURCE_GROUP: "<azure_resource_group>" 7
- 1
- Defines the default instance size that is used when a type is not defined in the workload.
- 2
- Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
- 3
- Specify the
AZURE_SUBNET_ID
value that you retrieved. - 4
- Specify the
AZURE_NSG_ID
value that you retrieved. - 5
- Optional: By default, this value is populated when you run the
KataConfig
CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID. - 6
- Specify the
AZURE_REGION
value you retrieved. - 7
- Specify the
AZURE_RESOURCE_GROUP
value you retrieved.
Click Save to apply the changes.
A config map is created for your cloud provider.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, click Save to apply the changes. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting the daemonset recreates the peer pods. It does not update the existing pods.
Verification
-
Navigate to Workloads
ConfigMaps to view the new config map.
3.2.2.3. Creating an SSH key secret
You must create an SSH key secret
object for Azure.
Procedure
- Log in to your OpenShift Container Platform cluster.
Generate an SSH key pair by running the following command:
$ ssh-keygen -f ./id_rsa -N ""
-
In the OpenShift Container Platform web console, navigate to Workloads
Secrets. - On the Secrets page, verify that you are in the openshift-sandboxed-containers-operator project.
- Click Create and select Key/value secret.
-
In the Secret name field, enter
ssh-key-secret
. -
In the Key field, enter
id_rsa.pub
. - In the Value field, paste your public SSH key.
Click Create.
The SSH key secret is created.
Delete the SSH keys you created:
$ shred -remove id_rsa.pub id_rsa
3.2.2.4. Creating a KataConfig custom resource
You must create a KataConfig
custom resource (CR) to install kata-remote
as a RuntimeClass
on your worker nodes.
The kata-remote
runtime class is installed on all worker nodes by default. If you want to install kata-remote
only on specific nodes, you can add labels to those nodes and then define the label in the KataConfig
CR.
OpenShift sandboxed containers installs kata-remote
as a secondary, optional runtime on the cluster and not as the primary runtime.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors might increase the reboot time:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
In the OpenShift Container Platform web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator.
- On the KataConfig tab, click Create KataConfig.
Enter the following details:
-
Name: Optional: The default name is
example-kataconfig
. -
Labels: Optional: Enter any relevant, identifying attributes to the
KataConfig
resource. Each label represents a key-value pair. - enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.
kataConfigPoolSelector. Optional: To install
kata-remote
on selected nodes, add a match expression for the labels on the selected nodes:- Expand the kataConfigPoolSelector area.
- In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
- Click Add matchExpressions.
- In the Key field, enter the label key the selector applies to.
-
In the Operator field, enter the key’s relationship to the label values. Valid operators are
In
,NotIn
,Exists
, andDoesNotExist
. - Expand the Values area and then click Add value.
-
In the Value field, enter
true
orfalse
for key label value.
-
logLevel: Define the level of log data retrieved for nodes with the
kata-remote
runtime class.
-
Name: Optional: The default name is
Click Create. The
KataConfig
CR is created and installs thekata-remote
runtime class on the worker nodes.Wait for the
kata-remote
installation to complete and the worker nodes to reboot before verifying the installation.
Verification
-
On the KataConfig tab, click the
KataConfig
CR to view its details. Click the YAML tab to view the
status
stanza.The
status
stanza contains theconditions
andkataNodes
keys. The value ofstatus.kataNodes
is an array of nodes, each of which lists nodes in a particular state ofkata-remote
installation. A message appears each time there is an update.Click Reload to refresh the YAML.
When all workers in the
status.kataNodes
array display the valuesinstalled
andconditions.InProgress: False
with no specified reason, thekata-remote
is installed on the cluster.
See KataConfig status messages for details.
3.2.2.4.1. Optional: Verifying the pod VM image
After kata-remote
is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.
Procedure
-
Navigate to Workloads
ConfigMaps. - Click the provider config map to view its details.
- Click the YAML tab.
Check the
status
stanza of the YAML file.If the
AZURE_IMAGE_ID
parameter is populated, the pod VM image was created successfully.
Troubleshooting
Retrieve the events log by running the following command:
$ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
Retrieve the job log by running the following command:
$ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.
3.2.2.5. Optional: Modifying the number of peer pod VMs per node
You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig
custom resource (CR).
Procedure
Check the current limit by running the following command:
$ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}'
Modify the
limit
attribute of thepeerpodConfig
CR by running the following command:$ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1
- 1
- Replace <value> with the limit you want to define.
3.2.2.6. Configuring workload objects
You deploy an OpenShift sandboxed containers workload by configuring kata-remote
as the runtime class for the following pod-templated objects:
-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
Do not deploy workloads in the openshift-sandboxed-containers-operator
namespace. Create a dedicated namespace for these resources.
You can define whether the workload should be deployed using the default instance size, which you defined in the config map, by adding an annotation to the YAML file.
If you do not want to define the instance size manually, you can add an annotation to use an automatic instance size, based on the memory available.
Prerequisites
- You have created a secret object for your provider.
- You have created a config map for your provider.
-
You have created a
KataConfig
custom resource (CR).
Procedure
-
In the OpenShift Container Platform web console, navigate to Workloads
workload type, for example, Pods. - On the workload type page, click an object to view its details.
- Click the YAML tab.
Add
spec.runtimeClassName: kata-remote
to the manifest of each pod-templated workload object as in the following example:apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ...
Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:
To use a manually defined instance size, add the following annotation:
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: Standard_B2als_v2 1 # ...
- 1
- Specify the instance size that you defined in the config map.
To use an automatic instance size, add the following annotations:
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ...
Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.
Click Save to apply the changes.
OpenShift Container Platform creates the workload object and begins scheduling it.
Verification
-
Inspect the
spec.runtimeClassName
field of a pod-templated object. If the value iskata-remote
, then the workload is running on OpenShift sandboxed containers, using peer pods.
3.2.3. Deploying workloads by using the command line
You can deploy OpenShift sandboxed containers workloads by using the command line.
3.2.3.1. Creating a secret
You must create a Secret
object on your OpenShift Container Platform cluster. The secret stores cloud provider credentials for creating the pod virtual machine (VM) image and peer pod instances. By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.
Prerequisites
- You have installed and configured the Azure CLI tool.
Procedure
Retrieve the Azure subscription ID:
$ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
Generate the RBAC content. This generates the client ID, client secret, and the tenant ID:
$ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query "{ client_id: appId, client_secret: password, tenant_id: tenant }
Example output:
{ "client_id": `AZURE_CLIENT_ID`, "client_secret": `AZURE_CLIENT_SECRET`, "tenant_id": `AZURE_TENANT_ID` }
-
Record the RBAC output to use in the
secret
object. Create a
peer-pods-secret.yaml
manifest file according to the following example:apiVersion: v1 kind: Secret metadata: name: peer-pods-secret namespace: openshift-sandboxed-containers-operator type: Opaque stringData: AZURE_CLIENT_ID: "<azure_client_id>" 1 AZURE_CLIENT_SECRET: "<azure_client_secret>" 2 AZURE_TENANT_ID: "<azure_tenant_id>" 3 AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>" 4
Create the
secret
object by applying the manifest:$ oc apply -f peer-pods-secret.yaml
If you update the peer pods secret, you must restart the peerpodconfig-ctrl-caa-daemon
DaemonSet to apply the changes.
After you update the secret, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting a daemon set recreates peer pods. It does not update existing pods.
3.2.3.2. Creating a config map
You must create a config map on your OpenShift Container Platform cluster for your cloud provider.
Procedure
Obtain the following values from your Azure instance:
Retrieve and record the Azure VNet name:
$ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
This value is used to retrieve the Azure subnet ID.
Retrieve and record the Azure subnet ID:
$ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
Retrieve and record the Azure network security group (NSG) ID:
$ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
Retrieve and record the Azure resource group:
$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
Retrieve and record the Azure region:
$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
Create a
peer-pods-cm.yaml
manifest according to the following example:apiVersion: v1 kind: ConfigMap metadata: name: peer-pods-cm namespace: openshift-sandboxed-containers-operator data: CLOUD_PROVIDER: "azure" VXLAN_PORT: "9000" AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1 AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5" 2 AZURE_SUBNET_ID: "<azure_subnet_id>" 3 AZURE_NSG_ID: "<azure_nsg_id>" 4 PROXY_TIMEOUT: "5m" DISABLECVM: "true" AZURE_IMAGE_ID: "<azure_image_id>" 5 AZURE_REGION: "<azure_region>" 6 AZURE_RESOURCE_GROUP: "<azure_resource_group>" 7
- 1
- Defines the default instance size that is used when a type is not defined in the workload.
- 2
- Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
- 3
- Specify the
AZURE_SUBNET_ID
value that you retrieved. - 4
- Specify the
AZURE_NSG_ID
value that you retrieved. - 5
- Optional: By default, this value is populated when you run the
KataConfig
CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID. - 6
- Specify the
AZURE_REGION
value you retrieved. - 7
- Specify the
AZURE_RESOURCE_GROUP
value you retrieved.
Apply the manifest to create a config map:
$ oc apply -f peer-pods-cm.yaml
A config map is created for your cloud provider.
If you update the peer pods config map, you must restart the peerpodconfig-ctrl-caa-daemon
daemonset to apply the changes.
After you update the config map, apply the manifest. Then restart the cloud-api-adaptor
pods by running the following command:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon -n openshift-sandboxed-containers-operator REBOOT="$(date)"
Restarting the daemonset recreates the peer pods. It does not update the existing pods.
3.2.3.3. Creating an SSH key secret
You must create an SSH key secret
object for Azure.
Procedure
- Log in to your OpenShift Container Platform cluster.
Generate an SSH key pair by running the following command:
$ ssh-keygen -f ./id_rsa -N ""
Create the
Secret
object by running the following command:$ oc create secret generic ssh-key-secret \ -n openshift-sandboxed-containers-operator \ --from-file=id_rsa.pub=./id_rsa.pub \ --from-file=id_rsa=./id_rsa
The SSH key secret is created.
Delete the SSH keys you created:
$ shred -remove id_rsa.pub id_rsa
3.2.3.4. Creating a KataConfig custom resource
You must create a KataConfig
custom resource (CR) to install kata-remote
as a runtime class on your worker nodes.
Creating the KataConfig
CR triggers the OpenShift sandboxed containers Operator to do the following:
-
Create a
RuntimeClass
CR namedkata-remote
with a default configuration. This enables users to configure workloads to usekata-remote
as the runtime by referencing the CR in theRuntimeClassName
field. This CR also specifies the resource overhead for the runtime.
OpenShift sandboxed containers installs kata-remote
as a secondary, optional runtime on the cluster and not as the primary runtime.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard disk drive rather than an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU and network.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
Create a
cluster-kataconfig.yaml
manifest file according to the following example:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: enablePeerPods: true logLevel: info
Optional: To install
kata-remote
on selected nodes, specify the node labels according to the following example:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1 # ...
- 1
- Specify the labels of the selected nodes.
Create the
KataConfig
CR:$ oc create -f cluster-kataconfig.yaml
The new
KataConfig
CR is created and installskata-remote
as a runtime class on the worker nodes.Wait for the
kata-remote
installation to complete and the worker nodes to reboot before verifying the installation.
Verification
Monitor the installation progress by running the following command:
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
When the status of all workers under
kataNodes
isinstalled
and the conditionInProgress
isFalse
without specifying a reason, thekata-remote
is installed on the cluster.
See KataConfig status messages for details.
3.2.3.4.1. Optional: Verifying the pod VM image
After kata-remote
is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.
Procedure
Obtain the config map you created for the peer pods:
$ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml
Check the
status
stanza of the YAML file.If the
AZURE_IMAGE_ID
parameter is populated, the pod VM image was created successfully.
Troubleshooting
Retrieve the events log by running the following command:
$ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
Retrieve the job log by running the following command:
$ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.
3.2.3.5. Optional: Modifying the number of peer pod VMs per node
You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig
custom resource (CR).
Procedure
Check the current limit by running the following command:
$ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ -o jsonpath='{.spec.limit}{"\n"}'
Modify the
limit
attribute of thepeerpodConfig
CR by running the following command:$ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \ --type merge --patch '{"spec":{"limit":"<value>"}}' 1
- 1
- Replace <value> with the limit you want to define.
3.2.3.6. Configuring workload objects
You deploy an OpenShift sandboxed containers workload by configuring kata-remote
as the runtime class for the following pod-templated objects:
-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
Do not deploy workloads in the openshift-sandboxed-containers-operator
namespace. Create a dedicated namespace for these resources.
You can define whether the workload should be deployed using the default instance size, which you defined in the config map, by adding an annotation to the YAML file.
If you do not want to define the instance size manually, you can add an annotation to use an automatic instance size, based on the memory available.
Prerequisites
- You have created a secret object for your provider.
- You have created a config map for your provider.
-
You have created a
KataConfig
custom resource (CR).
Procedure
Add
spec.runtimeClassName: kata-remote
to the manifest of each pod-templated workload object as in the following example:apiVersion: v1 kind: <object> # ... spec: runtimeClassName: kata-remote # ...
Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:
To use a manually defined instance size, add the following annotation:
apiVersion: v1 kind: <object> metadata: annotations: io.katacontainers.config.hypervisor.machine_type: Standard_B2als_v2 1 # ...
- 1
- Specify the instance size that you defined in the config map.
To use an automatic instance size, add the following annotations:
apiVersion: v1 kind: <Pod> metadata: annotations: io.katacontainers.config.hypervisor.default_vcpus: <vcpus> io.katacontainers.config.hypervisor.default_memory: <memory> # ...
Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.
Apply the changes to the workload object by running the following command:
$ oc apply -f <object.yaml>
OpenShift Container Platform creates the workload object and begins scheduling it.
Verification
-
Inspect the
spec.runtimeClassName
field of a pod-templated object. If the value iskata-remote
, then the workload is running on OpenShift sandboxed containers, using peer pods.