Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 3. Deploying OpenShift sandboxed containers workloads
You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI (oc
). Before installing the OpenShift sandboxed containers Operator, you must prepare your OpenShift Container Platform cluster.
3.1. Prerequisites
Before you install OpenShift sandboxed containers, ensure that your OpenShift Container Platform cluster meets the following requirements:
Your cluster must be installed on on-premise bare-metal infrastructure with Red Hat Enterprise Linux CoreOS (RHCOS) workers. You can use any installation method including user-provisioned, installer-provisioned, or assisted installer to deploy your cluster.
Note- OpenShift sandboxed containers only supports RHCOS worker nodes. RHEL nodes are not supported.
- Nested virtualization is not supported.
You can install OpenShift sandboxed containers on Amazon Web Services (AWS) bare-metal instances. Bare-metal instances offered by other cloud providers are not supported.
ImportantInstalling OpenShift sandboxed containers on AWS bare-metal instances is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
3.1.1. Resource requirements for OpenShift sandboxed containers
OpenShift sandboxed containers lets users run workloads on their OpenShift Container Platform clusters inside a sandboxed runtime (Kata). Each pod is represented by a virtual machine (VM). Each VM runs in a QEMU process and hosts a kata-agent
process that acts as a supervisor for managing container workloads, and the processes running in those containers. Two additional processes add more overhead:
-
containerd-shim-kata-v2
is used to communicate with the pod. -
virtiofsd
handles host file system access on behalf of the guest.
Each VM is configured with a default amount of memory. Additional memory is hot-plugged into the VM for containers that explicitly request memory.
A container running without a memory resource consumes free memory until the total memory used by the VM reaches the default allocation. The guest and its I/O buffers also consume memory.
If a container is given a specific amount of memory, then that memory is hot-plugged into the VM before the container starts.
When a memory limit is specified, the workload is terminated if it consumes more memory than the limit. If no memory limit is specified, the kernel running on the VM might run out of memory. If the kernel runs out of memory, it might terminate other processes on the VM.
Default memory sizes
The following table lists some the default values for resource allocation.
Resource | Value |
---|---|
Memory allocated by default to a virtual machine | 2Gi |
Guest Linux kernel memory usage at boot | ~110Mi |
Memory used by the QEMU process (excluding VM memory) | ~30Mi |
Memory used by the | ~10Mi |
Memory used by the | ~20Mi |
File buffer cache data after running | ~300Mi* [1] |
File buffers appear and are accounted for in multiple locations:
- In the guest where it appears as file buffer cache.
-
In the
virtiofsd
daemon that maps allowed user-space file I/O operations. - In the QEMU process as guest memory.
Total memory usage is properly accounted for by the memory utilization metrics, which only count that memory once.
Pod overhead describes the amount of system resources that a pod on a node uses. You can get the current pod overhead for the Kata runtime by using oc describe runtimeclass kata
as shown below.
Example
$ oc describe runtimeclass kata
Example output
kind: RuntimeClass apiVersion: node.k8s.io/v1 metadata: name: kata overhead: podFixed: memory: "500Mi" cpu: "500m"
You can change the pod overhead by changing the spec.overhead
field for a RuntimeClass
. For example, if the configuration that you run for your containers consumes more than 350Mi of memory for the QEMU process and guest kernel data, you can alter the RuntimeClass
overhead to suit your needs.
The specified default overhead values are supported by Red Hat. Changing default overhead values is not supported and can result in technical issues.
When performing any kind of file system I/O in the guest, file buffers are allocated in the guest kernel. The file buffers are also mapped in the QEMU process on the host, as well as in the virtiofsd
process.
For example, if you use 300Mi of file buffer cache in the guest, both QEMU and virtiofsd
appear to use 300Mi additional memory. However, the same memory is being used in all three cases. In other words, the total memory usage is only 300Mi, mapped in three different places. This is correctly accounted for when reporting the memory utilization metrics.
Additional resources
3.1.2. Checking whether cluster nodes are eligible to run OpenShift sandboxed containers
Before running OpenShift sandboxed containers, you can check whether the nodes in your cluster are eligible to run Kata containers. Some cluster nodes might not comply with sandboxed containers' minimum requirements. The most common reason for node ineligibility is the lack of virtualization support on the node. If you attempt to run sandboxed workloads on ineligible nodes, you will experience errors. You can use the Node Feature Discovery (NFD) Operator and a NodeFeatureDiscovery
resource to automatically check node eligibility.
If you want to install the Kata runtime on only selected worker nodes that you know are eligible, apply the feature.node.kubernetes.io/runtime.kata=true
label to the selected nodes and set checkNodeEligibility: true
in the KataConfig
resource.
Alternatively, to install the Kata runtime on all worker nodes, set checkNodeEligibility: false
in the KataConfig
resource.
In both these scenarios, you do not need to create the NodeFeatureDiscovery
resource. You should only apply the feature.node.kubernetes.io/runtime.kata=true
label manually if you are sure that the node is eligible to run Kata containers.
The following procedure applies the feature.node.kubernetes.io/runtime.kata=true
label to all eligible nodes and configures the KataConfig
resource to check for node eligibility.
Prerequisites
-
Install the OpenShift CLI (
oc
). -
Log in as a user with
cluster-admin
privileges. - Install the Node Feature Discovery (NFD) Operator.
Procedure
Create a
NodeFeatureDiscovery
resource to detect node capabilities suitable for running Kata containers:Save the following YAML in the
nfd.yaml
file:apiVersion: nfd.openshift.io/v1 kind: NodeFeatureDiscovery metadata: name: nfd-kata namespace: openshift-nfd spec: operand: image: quay.io/openshift/origin-node-feature-discovery:4.10 imagePullPolicy: Always servicePort: 12000 workerConfig: configData: | sources: custom: - name: "feature.node.kubernetes.io/runtime.kata" matchOn: - cpuId: ["SSE4", "VMX"] loadedKMod: ["kvm", "kvm_intel"] - cpuId: ["SSE4", "SVM"] loadedKMod: ["kvm", "kvm_amd"]
Create the
NodeFeatureDiscovery
custom resource (CR):$ oc create -f nfd.yaml
Example output
nodefeaturediscovery.nfd.openshift.io/nfd-kata created
A
feature.node.kubernetes.io/runtime.kata=true
label is applied to all qualifying worker nodes.
Set the
checkNodeEligibility
field totrue
in theKataConfig
resource to enable the feature, for example:Save the following YAML in the
kata-config.yaml
file:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: example-kataconfig spec: checkNodeEligibility: true
Create the
KataConfig
CR:$ oc create -f kata-config.yaml
Example output
kataconfig.kataconfiguration.openshift.io/example-kataconfig created
Verification
Verify that qualifying nodes in the cluster have the correct label applied:
$ oc get nodes --selector='feature.node.kubernetes.io/runtime.kata=true'
Example output
NAME STATUS ROLES AGE VERSION compute-3.example.com Ready worker 4h38m v1.23.3+e419edf compute-2.example.com Ready worker 4h35m v1.23.3+e419edf
Additional resources
- For more information about installing the Node Feature Discovery (NFD) Operator, see Installing NFD.
3.2. Deploying OpenShift sandboxed containers workloads using the web console
You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig
custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata
as the runtimeClassName
to the workload YAML file.
3.2.1. Installing the OpenShift sandboxed containers Operator using the web console
You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console.
Prerequisites
- You have OpenShift Container Platform 4.10 installed.
-
You have access to the cluster as a user with the
cluster-admin
role.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
OperatorHub. -
In the Filter by keyword field, type
OpenShift sandboxed containers
. - Select the OpenShift sandboxed containers tile.
- Read the information about the Operator and click Install.
On the Install Operator page:
- Select stable-1.2 from the list of available Update Channel options.
Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory
openshift-sandboxed-containers-operator
namespace. If this namespace does not yet exist, it is automatically created.NoteAttempting to install the OpenShift sandboxed containers Operator in a namespace other than
openshift-sandboxed-containers-operator
causes the installation to fail.- Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
- Click Install.
The OpenShift sandboxed containers Operator is now installed on your cluster.
Verification
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.
3.2.2. Creating the KataConfig custom resource in the web console
You must create one KataConfig
custom resource (CR) to enable installing kata
as a RuntimeClass
on your cluster nodes.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard drive rather than on an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU or network.
Prerequisites
- You have installed OpenShift Container Platform 4.10 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
Kata is installed on all worker nodes by default. If you want to install kata
as a RuntimeClass
only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig
CR when you create it.
Procedure
-
From the Administrator perspective in the web console, navigate to Operators
Installed Operators. - Select the OpenShift sandboxed containers Operator from the list of operators.
- In the KataConfig tab, click Create KataConfig.
-
In the Create KataConfig page, select to configure the
KataConfig
CR via YAML view. Copy and paste the following manifest into the YAML view:
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataMonitorImage: registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel8:1.2.0
If you want to install
kata
as aRuntimeClass
only on selected nodes, include the label in the manifest:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataMonitorImage: registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel8:1.2.0 kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
- 1
- Labels in
kataConfigPoolSelector
only support single values;nodeSelector
syntax is not supported.
- Click Create.
The new KataConfig
CR is created and begins to install kata
as a RuntimeClass
on the worker nodes. Wait for the kata
installation to complete and the worker nodes to reboot before continuing to the next step.
OpenShift sandboxed containers installs Kata only as a secondary, optional runtime on the cluster and not as the primary runtime.
Verification
-
In the KataConfig tab, select the new
KataConfig
CR. - In the KataConfig page, select the YAML tab.
Monitor the installationStatus field in the status.
A message appears each time there is an update. Click Reload to view the updated
KataConfig
CR.Once the value of Completed nodes equals the number of worker or labeled nodes, the installation is complete. The status also contains a list of nodes where the installation is completed.
3.2.3. Deploying a workload in a sandboxed container using the web console
OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.
To deploy a pod-templated workload in a sandboxed container, you must manually add kata
as the runtimeClassName
to the workload YAML file.
Prerequisites
- You have installed OpenShift Container Platform 4.10 on your cluster.
-
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
-
You have created a
KataConfig
custom resource (CR).
Procedure
- From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
- In the workload page, click to create the workload.
In the YAML file for the workload, in the spec field where the container is listed, add
runtimeClassName: kata
.Example for
Pod
objectapiVersion: v1 kind: Pod metadata: name: hello-openshift labels: app: hello-openshift spec: runtimeClassName: kata containers: - name: hello-openshift image: quay.io/openshift/origin-hello-openshift ports: - containerPort: 8888 securityContext: privileged: false allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault
- Click Save.
OpenShift Container Platform creates the workload and begins scheduling it.
3.3. Deploying OpenShift sandboxed containers workloads using the CLI
You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig
custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata
as the runtimeClassName
to the workload YAML file.
3.3.1. Installing the OpenShift sandboxed containers Operator using the CLI
You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI.
Prerequisites
- You have OpenShift Container Platform 4.10 installed on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. You have subscribed to the OpenShift sandboxed containers catalog.
NoteSubscribing to the OpenShift sandboxed containers catalog provides
openshift-sandboxed-containers-operator
namespace access to the OpenShift sandboxed containers Operator.
Procedure
Create the
Namespace
object for the OpenShift sandboxed containers Operator.Create a
Namespace
object YAML file that contains the following manifest:apiVersion: v1 kind: Namespace metadata: name: openshift-sandboxed-containers-operator
Create the
Namespace
object:$ oc create -f Namespace.yaml
Create the
OperatorGroup
object for the OpenShift sandboxed containers Operator.Create an
OperatorGroup
object YAML file that contains the following manifest:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: targetNamespaces: - openshift-sandboxed-containers-operator
Create the
OperatorGroup
object:$ oc create -f OperatorGroup.yaml
Create the
Subscription
object to subscribe theNamespace
to the OpenShift sandboxed containers Operator.Create a
Subscription
object YAML file that contains the following manifest:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-sandboxed-containers-operator namespace: openshift-sandboxed-containers-operator spec: channel: "stable-1.2" installPlanApproval: Automatic name: sandboxed-containers-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: sandboxed-containers-operator.v1.2.2
Create the
Subscription
object:$ oc create -f Subscription.yaml
The OpenShift sandboxed containers Operator is now installed on your cluster.
All the object file names listed above are suggestions. You can create the object YAML files using other names.
Verification
Ensure that the Operator is correctly installed:
$ oc get csv -n openshift-sandboxed-containers-operator
Example output
NAME DISPLAY VERSION REPLACES PHASE openshift-sandboxed-containers openshift-sandboxed-containers-operator 1.2.2 1.2.1 Succeeded
Additional resources
3.3.2. Creating the KataConfig custom resource using the CLI
You must create one KataConfig
custom resource (CR) to install kata
as a RuntimeClass
on your nodes. Creating the KataConfig
CR triggers the OpenShift sandboxed containers Operator to do the following:
-
Install the needed RHCOS extensions, such as QEMU and
kata-containers
, on your RHCOS node. -
Ensure that the CRI-O runtime is configured with the correct
kata
runtime handlers. -
Create a
RuntimeClass
CR namedkata
with a default configuration. This enables users to configure workloads to usekata
as the runtime by referencing the CR in theRuntimeClassName
field. This CR also specifies the resource overhead for the runtime.
Kata is installed on all worker nodes by default. If you want to install kata
as a RuntimeClass
only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig
CR when you create it.
Prerequisites
- You have installed OpenShift Container Platform 4.10 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
Creating the KataConfig
CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:
- A larger OpenShift Container Platform deployment with a greater number of worker nodes.
- Activation of the BIOS and Diagnostics utility.
- Deployment on a hard drive rather than on an SSD.
- Deployment on physical nodes such as bare metal, rather than on virtual nodes.
- A slow CPU or network.
Procedure
Create a YAML file with the following manifest:
apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataMonitorImage: registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel8:1.2.0
(Optional) If you want to install
kata
as aRuntimeClass
only on selected nodes, create a YAML file that includes the label in the manifest:apiVersion: kataconfiguration.openshift.io/v1 kind: KataConfig metadata: name: cluster-kataconfig spec: kataMonitorImage: registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel8:1.2.0 kataConfigPoolSelector: matchLabels: <label_key>: '<label_value>' 1
- 1
- Labels in
kataConfigPoolSelector
only support single values;nodeSelector
syntax is not supported.
Create the
KataConfig
resource:$ oc create -f <file name>.yaml
The new KataConfig
CR is created and begins to install kata
as a RuntimeClass
on the worker nodes. Wait for the `kata`installation to complete and the worker nodes to reboot before continuing to the next step.
OpenShift sandboxed containers installs Kata only as a secondary, optional runtime on the cluster and not as the primary runtime.
Verification
Monitor the installation progress:
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
Once the value of Is In Progress appears as
false
, the installation is complete.
Additional resources
3.3.3. Deploying a workload in a sandboxed container using the CLI
OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.
To deploy a pod-templated workload in a sandboxed container, you must add kata
as the runtimeClassName
to the workload YAML file.
Prerequisites
- You have installed OpenShift Container Platform 4.10 on your cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the OpenShift sandboxed containers Operator.
-
You have created a
KataConfig
custom resource (CR).
Procedure
Add
runtimeClassName: kata
to any pod-templated object:-
Pod
objects -
ReplicaSet
objects -
ReplicationController
objects -
StatefulSet
objects -
Deployment
objects -
DeploymentConfig
objects
-
Example for Pod
objects
apiVersion: v1 kind: Pod metadata: name: hello-openshift labels: app: hello-openshift spec: runtimeClassName: kata containers: - name: hello-openshift image: quay.io/openshift/origin-hello-openshift ports: - containerPort: 8888 securityContext: privileged: false allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault
OpenShift Container Platform creates the workload and begins scheduling it.
Verification
-
Inspect the
runtimeClassName
field on a pod-templated object. If theruntimeClassName
iskata
, then the workload is running on a OpenShift sandboxed containers.
3.4. Additional resources
- The OpenShift sandboxed containers Operator is supported in a restricted network environment. For more information, Using Operator Lifecycle Manager on restricted networks.
- When using a disconnected cluster on a restricted network, you must configure proxy support in Operator Lifecycle Manager to access the OperatorHub. Using a proxy allows the cluster to fetch the OpenShift sandboxed containers Operator.