Chapter 2. Deploying OpenShift sandboxed containers workloads


You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI (oc). Before installing the OpenShift sandboxed containers Operator, you must prepare your Red Hat OpenShift cluster.

2.1. Prerequisites

Before you install OpenShift sandboxed containers, ensure that your Red Hat OpenShift cluster meets the following requirements:

  • Your cluster must be installed on on-premise bare-metal infrastructure with Red Hat Enterprise Linux CoreOS (RHCOS) workers. You can use any installation method including user-provisioned, installer-provisioned, or assisted installer to deploy your cluster.

    Note
    • OpenShift sandboxed containers only supports RHCOS worker nodes. RHEL nodes are not supported.
    • Nested virtualization is not supported.
  • You can install OpenShift sandboxed containers on Amazon Web Services (AWS) bare-metal instances. Bare-metal instances offered by other cloud providers are not supported.

2.1.1. Resource requirements for OpenShift sandboxed containers

OpenShift sandboxed containers lets users run workloads on their Red Hat OpenShift clusters inside a sandboxed runtime (Kata). Each pod is represented by a virtual machine (VM). Each VM runs in a QEMU process and hosts a kata-agent process that acts as a supervisor for managing container workloads, and the processes running in those containers. Two additional processes add more overhead:

  • containerd-shim-kata-v2 is used to communicate with the pod.
  • virtiofsd handles host file system access on behalf of the guest.

Each VM is configured with a default amount of memory. Additional memory is hot-plugged into the VM for containers that explicitly request memory.

A container running without a memory resource consumes free memory until the total memory used by the VM reaches the default allocation. The guest and its I/O buffers also consume memory.

If a container is given a specific amount of memory, then that memory is hot-plugged into the VM before the container starts.

When a memory limit is specified, the workload is terminated if it consumes more memory than the limit. If no memory limit is specified, the kernel running on the VM might run out of memory. If the kernel runs out of memory, it might terminate other processes on the VM.

Default memory sizes

The following table lists some the default values for resource allocation.

ResourceValue

Memory allocated by default to a virtual machine

2Gi

Guest Linux kernel memory usage at boot

~110Mi

Memory used by the QEMU process (excluding VM memory)

~30Mi

Memory used by the virtiofsd process (excluding VM I/O buffers)

~10Mi

Memory used by the containerd-shim-kata-v2 process

~20Mi

File buffer cache data after running dnf install on Fedora

~300Mi* [1]

File buffers appear and are accounted for in multiple locations:

  • In the guest where it appears as file buffer cache.
  • In the virtiofsd daemon that maps allowed user-space file I/O operations.
  • In the QEMU process as guest memory.
Note

Total memory usage is properly accounted for by the memory utilization metrics, which only count that memory once.

Pod overhead describes the amount of system resources that a pod on a node uses. You can get the current pod overhead for the Kata runtime by using oc describe runtimeclass kata as shown below.

Example

$ oc describe runtimeclass kata

Example output

kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
  name: kata
overhead:
  podFixed:
    memory: "500Mi"
    cpu: "500m"

You can change the pod overhead by changing the spec.overhead field for a RuntimeClass. For example, if the configuration that you run for your containers consumes more than 350Mi of memory for the QEMU process and guest kernel data, you can alter the RuntimeClass overhead to suit your needs.

Note

The specified default overhead values are supported by Red Hat. Changing default overhead values is not supported and can result in technical issues.

When performing any kind of file system I/O in the guest, file buffers are allocated in the guest kernel. The file buffers are also mapped in the QEMU process on the host, as well as in the virtiofsd process.

For example, if you use 300Mi of file buffer cache in the guest, both QEMU and virtiofsd appear to use 300Mi additional memory. However, the same memory is being used in all three cases. In other words, the total memory usage is only 300Mi, mapped in three different places. This is correctly accounted for when reporting the memory utilization metrics.

2.1.2. Checking whether cluster nodes are eligible to run OpenShift sandboxed containers

Before running OpenShift sandboxed containers, you can check whether the nodes in your cluster are eligible to run Kata containers. Some cluster nodes might not comply with sandboxed containers' minimum requirements. The most common reason for node ineligibility is the lack of virtualization support on the node. If you attempt to run sandboxed workloads on ineligible nodes, you will experience errors. You can use the Node Feature Discovery (NFD) Operator and a NodeFeatureDiscovery resource to automatically check node eligibility.

Note

If you want to install the Kata runtime on only selected worker nodes that you know are eligible, apply the feature.node.kubernetes.io/runtime.kata=true label to the selected nodes and set checkNodeEligibility: true in the KataConfig resource.

Alternatively, to install the Kata runtime on all worker nodes, set checkNodeEligibility: false in the KataConfig resource.

In both these scenarios, you do not need to create the NodeFeatureDiscovery resource. You should only apply the feature.node.kubernetes.io/runtime.kata=true label manually if you are sure that the node is eligible to run Kata containers.

The following procedure applies the feature.node.kubernetes.io/runtime.kata=true label to all eligible nodes and configures the KataConfig resource to check for node eligibility.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.
  • Install the Node Feature Discovery (NFD) Operator.

Procedure

  1. Create a NodeFeatureDiscovery resource to detect node capabilities suitable for running Kata containers:

    1. Save the following YAML in the nfd.yaml file:

      apiVersion: nfd.openshift.io/v1
      kind: NodeFeatureDiscovery
      metadata:
        name: nfd-kata
        namespace: openshift-nfd
      spec:
        operand:
          image: quay.io/openshift/origin-node-feature-discovery:4.10
          imagePullPolicy: Always
          servicePort: 12000
        workerConfig:
          configData: |
            sources:
               custom:
                 - name: "feature.node.kubernetes.io/runtime.kata"
                   matchOn:
                     - cpuId: ["SSE4", "VMX"]
                       loadedKMod: ["kvm", "kvm_intel"]
                     - cpuId: ["SSE4", "SVM"]
                       loadedKMod: ["kvm", "kvm_amd"]
    2. Create the NodeFeatureDiscovery custom resource (CR):

      $ oc create -f nfd.yaml

      Example output

      nodefeaturediscovery.nfd.openshift.io/nfd-kata created

      A feature.node.kubernetes.io/runtime.kata=true label is applied to all qualifying worker nodes.

  2. Set the checkNodeEligibility field to true in the KataConfig resource to enable the feature, for example:

    1. Save the following YAML in the kata-config.yaml file:

      apiVersion: kataconfiguration.openshift.io/v1
      kind: KataConfig
      metadata:
        name: example-kataconfig
      spec:
        checkNodeEligibility: true
    2. Create the KataConfig CR:

      $ oc create -f kata-config.yaml

      Example output

      kataconfig.kataconfiguration.openshift.io/example-kataconfig created

Verification

  • Verify that qualifying nodes in the cluster have the correct label applied:

    $ oc get nodes --selector='feature.node.kubernetes.io/runtime.kata=true'

    Example output

    NAME                           STATUS                     ROLES    AGE     VERSION
    compute-3.example.com          Ready                      worker   4h38m   v1.25.0
    compute-2.example.com          Ready                      worker   4h35m   v1.25.0

Additional resources

  • For more information about installing the Node Feature Discovery (NFD) Operator, see Installing NFD.

2.2. Deploying OpenShift sandboxed containers workloads using the web console

You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata as the runtimeClassName to the workload YAML file.

2.2.1. Installing the OpenShift sandboxed containers Operator using the web console

You can install the OpenShift sandboxed containers Operator from the Red Hat OpenShift web console.

Prerequisites

  • You have Red Hat OpenShift 4.13 installed.
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. From the Administrator perspective in the web console, navigate to Operators OperatorHub.
  2. In the Filter by keyword field, type OpenShift sandboxed containers.
  3. Select the OpenShift sandboxed containers tile.
  4. Read the information about the Operator and click Install.
  5. On the Install Operator page:

    1. Select stable from the list of available Update Channel options.
    2. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

      Note

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

    3. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
  6. Click Install.

The OpenShift sandboxed containers Operator is now installed on your cluster.

Verification

  1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
  2. Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.

2.2.2. Creating the KataConfig custom resource in the web console

You must create one KataConfig custom resource (CR) to enable installing kata as a RuntimeClass on your cluster nodes.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger Red Hat OpenShift deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Note

Kata is installed on all worker nodes by default. If you want to install kata as a RuntimeClass only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig CR when you create it.

Procedure

  1. From the Administrator perspective in the web console, navigate to Operators Installed Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. In the KataConfig tab, click Create KataConfig.
  4. In the Create KataConfig page, enter the following details:

    • Name: Enter a name for the KataConfig resource. By default, the name is defined as example-kataconfig.
    • Labels (Optional): Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
    • checkNodeEligibility (Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
    • enablePeerPods (For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment.
    • kataConfigPoolSelector: By default, kata is installed as a RuntimeClass on all nodes. If you want to install kata as a RuntimeClass only on selected nodes, you must add a matchExpression:

      1. Expand the kataConfigPoolSelector area.
      2. In the kataConfigPoolSelector, expand matchExpressions. This is a list of label selector requirements.
      3. Click Add matchExpressions.
      4. In the key field, add the label key the selector applies to.
      5. In the operator field, add the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
      6. Expand the values area, and then click Add value.
      7. In the Value field, enter true or false for key label value.
    • logLevel: Define the level of log data retrieved for nodes running kata as a RuntimeClass. For more information, see "Collecting OpenShift sandboxed containers data".
  5. Click Create.

The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Wait for the kata installation to complete and the worker nodes to reboot before continuing to the next step.

Important

OpenShift sandboxed containers installs kata only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  1. In the KataConfig tab, select the new KataConfig CR.
  2. In the KataConfig page, select the YAML tab.
  3. Monitor the installationStatus field in the status.

    A message appears each time there is an update. Click Reload to view the updated KataConfig CR.

    Once the value of Completed nodes equals the number of worker or labeled nodes, the installation is complete. The status also contains a list of nodes where the installation is completed.

2.2.3. Deploying a workload in a sandboxed container using the web console

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload in a sandboxed container, you must manually add kata as the runtimeClassName to the workload YAML file.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a KataConfig custom resource (CR).

Procedure

  1. From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
  2. In the workload page, click to create the workload.
  3. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata.

    Example for Pod object

    apiVersion: v1
    kind: Pod
    metadata:
      name: hello-openshift
      labels:
        app: hello-openshift
    spec:
      runtimeClassName: kata
      containers:
        - name: hello-openshift
          image: quay.io/openshift/origin-hello-openshift
          ports:
            - containerPort: 8888
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

  4. Click Save.

Red Hat OpenShift creates the workload and begins scheduling it.

2.3. Deploying OpenShift sandboxed containers workloads using the CLI

You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata as the runtimeClassName to the workload YAML file.

2.3.1. Installing the OpenShift sandboxed containers Operator using the CLI

You can install the OpenShift sandboxed containers Operator using the Red Hat OpenShift CLI.

Prerequisites

  • You have Red Hat OpenShift 4.13 installed on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have subscribed to the OpenShift sandboxed containers catalog.

    Note

    Subscribing to the OpenShift sandboxed containers catalog provides openshift-sandboxed-containers-operator namespace access to the OpenShift sandboxed containers Operator.

Procedure

  1. Create the Namespace object for the OpenShift sandboxed containers Operator.

    1. Create a Namespace object YAML file that contains the following manifest:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-sandboxed-containers-operator
    2. Create the Namespace object:

      $ oc create -f Namespace.yaml
  2. Create the OperatorGroup object for the OpenShift sandboxed containers Operator.

    1. Create an OperatorGroup object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        targetNamespaces:
        - openshift-sandboxed-containers-operator
    2. Create the OperatorGroup object:

      $ oc create -f OperatorGroup.yaml
  3. Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator.

    1. Create a Subscription object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: sandboxed-containers-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: sandboxed-containers-operator.v1.4.1
    2. Create the Subscription object:

      $ oc create -f Subscription.yaml

The OpenShift sandboxed containers Operator is now installed on your cluster.

Note

All the object file names listed above are suggestions. You can create the object YAML files using other names.

Verification

  • Ensure that the Operator is correctly installed:

    $ oc get csv -n openshift-sandboxed-containers-operator

    Example output

    NAME                             DISPLAY                                  VERSION  REPLACES     PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.4.1    1.4.0        Succeeded

2.3.2. Creating the KataConfig custom resource using the CLI

You must create one KataConfig custom resource (CR) to install kata as a RuntimeClass on your nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following:

  • Install the needed RHCOS extensions, such as QEMU and kata-containers, on your RHCOS node.
  • Ensure that the CRI-O runtime is configured with the correct runtime handlers.
  • Create a RuntimeClass CR named kata with a default configuration. This enables users to configure workloads to use kata as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime.
Note

Kata is installed on all worker nodes by default. If you want to install kata as a RuntimeClass only on specific nodes, you can add labels to those nodes, and then define the label in the KataConfig CR when you create it.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger Red Hat OpenShift deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Procedure

  1. Create a YAML file with the following manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      checkNodeEligibility: false 1
      logLevel: info
    1
    Set`checkNodeEligibility` to true to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
  2. (Optional) If you want to install kata as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      checkNodeEligibility: false
      logLevel: info
      kataConfigPoolSelector:
        matchLabels:
          <label_key>: '<label_value>' 1
    1
    Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported.
  3. Create the KataConfig resource:

    $ oc create -f cluster-kataconfig.yaml

The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Wait for the kata installation to complete and the worker nodes to reboot before continuing to the next step.

Important

OpenShift sandboxed containers installs kata only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  • Monitor the installation progress:

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

    Once the value of Is In Progress appears as false, the installation is complete.

2.3.3. Deploying a workload in a sandboxed container using the CLI

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload in a sandboxed container, you must add kata as the runtimeClassName to the workload YAML file.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a KataConfig custom resource (CR).

Procedure

  • Add runtimeClassName: kata to any pod-templated object:

    • Pod objects
    • ReplicaSet objects
    • ReplicationController objects
    • StatefulSet objects
    • Deployment objects
    • DeploymentConfig objects

    .Example for Pod object

    apiVersion: v1
    kind: Pod
    metadata:
      name: hello-openshift
      labels:
        app: hello-openshift
    spec:
      runtimeClassName: kata
      containers:
        - name: hello-openshift
          image: quay.io/openshift/origin-hello-openshift
          ports:
            - containerPort: 8888
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

Red Hat OpenShift creates the workload and begins scheduling it.

Verification

  • Inspect the runtimeClassName field on a pod-templated object. If the runtimeClassName is kata, then the workload is running on OpenShift sandboxed containers.

2.4. Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.