Sandboxed Containers Support for OpenShift

The OpenShift sandboxed containers Operator now deploys an osc-monitor daemon set. This enables the collection of metrics specific to workloads running in sandboxed containers, including metrics about the hypervisor and the Guest OS instances. In addition, a pre-configured dashboard provides insights into OpenShift sandboxed containers components, such as the total number of lightweight VMs enabled in a cluster and the CPU and memory consumption per VM. All metrics, as well as the dashboard, are available in the web console. For more information, see Monitoring OpenShift sandboxed containers.

Administrators can now collect enhanced logs for OpenShift sandboxed containers runtime components. Enhanced logs are available when the CRI-O log level is set to debug. These logs are collected by the must-gather tool, or can be viewed in the node journal. For more information, see Enabling debug logs for OpenShift sandboxed containers.

Administrators can now check the eligibility of cluster nodes to run OpenShift sandboxed containers. This feature uses the Node Feature Discovery (NFD) Operator to detect node capabilities. Eligible nodes are labeled with feature.node.kubernetes.io/runtime.kata, and the OpenShift sandboxed containers Operator uses this label to select candidate nodes for installation.

The administrator must deploy the NFD Operator to use this feature, create a specific NodeFeatureDiscovery custom resource, and enable checkNodeEligibility when creating the KataConfig custom resource. For more information, see Checking the eligibility of cluster nodes to run OpenShift sandboxed containers.

Users can now run OpenShift sandboxed containers on clusters with OpenShift Virtualization when VMs are configured correctly. For more information, see Using OpenShift sandboxed containers with OpenShift Virtualization

Users can now install OpenShift sandboxed containers on AWS bare-metal clusters. This feature is in Technology Preview and not fully supported. For more information, see Understanding OpenShift sandboxed containers.

Issued: 2022-07-26

OpenShift sandboxed containers release 1.2.2 is now available. This advisory contains an update for OpenShift sandboxed containers with bug fixes.

The list of bug fixes included in the update is documented in the RHSA-2022:5725 advisory.

Issued: 2022-04-21

OpenShift sandboxed containers release 1.2.1 is now available. This advisory contains an update for OpenShift sandboxed containers with bug fixes.

The list of bug fixes included in the update is documented in the RHSA-2022:1508 advisory.

Issued: 2022-03-14

OpenShift sandboxed containers release 1.2.0 is now available. This advisory contains an update for OpenShift sandboxed containers with enhancements, security updates, and bug fixes.

The list of bug fixes included in the update is documented in the RHSA-2022:0855 advisory.

The OpenShift sandboxed containers Operator encapsulates all of the components from Kata containers. It manages installation, lifecycle, and configuration tasks.

The OpenShift sandboxed containers Operator is packaged in the Operator bundle format as two container images. The bundle image contains metadata and is required to make the operator OLM-ready. The second container image contains the actual controller that monitors and manages the KataConfig resource.

The OpenShift sandboxed containers Operator is based on the Red Hat Enterprise Linux CoreOS (RHCOS) extensions concept. Red Hat Enterprise Linux CoreOS (RHCOS) extensions are a mechanism to install optional OpenShift Container Platform software. The OpenShift sandboxed containers Operator uses this mechanism to deploy sandboxed containers on a cluster.

The sandboxed containers RHCOS extension contains RPMs for Kata, QEMU, and its dependencies. You can enable them by using the MachineConfig resources that the Machine Config Operator provides.

You can use OpenShift sandboxed containers on clusters with OpenShift Virtualization.

To run OpenShift Virtualization and OpenShift sandboxed containers at the same time, you must enable VMs to migrate, so that they do not block node reboots. Configure the following parameters on your VM:

OpenShift sandboxed containers lets users run workloads on their OpenShift Container Platform clusters inside a sandboxed runtime (Kata). Each pod is represented by a virtual machine (VM). Each VM runs in a QEMU process and hosts a kata-agent process that acts as a supervisor for managing container workloads, and the processes running in those containers. Two additional processes add more overhead:

Each VM is configured with a default amount of memory. Additional memory is hot-plugged into the VM for containers that explicitly request memory.

A container running without a memory resource consumes free memory until the total memory used by the VM reaches the default allocation. The guest and its I/O buffers also consume memory.

If a container is given a specific amount of memory, then that memory is hot-plugged into the VM before the container starts.

When a memory limit is specified, the workload is terminated if it consumes more memory than the limit. If no memory limit is specified, the kernel running on the VM might run out of memory. If the kernel runs out of memory, it might terminate other processes on the VM.

File buffers appear and are accounted for in multiple locations:

Pod overhead describes the amount of system resources that a pod on a node uses. You can get the current pod overhead for the Kata runtime by using oc describe runtimeclass kata as shown below.

$ oc describe runtimeclass kata

kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
  name: kata
overhead:
  podFixed:
    memory: "500Mi"
    cpu: "500m"

You can change the pod overhead by changing the spec.overhead field for a RuntimeClass. For example, if the configuration that you run for your containers consumes more than 350Mi of memory for the QEMU process and guest kernel data, you can alter the RuntimeClass overhead to suit your needs.

When performing any kind of file system I/O in the guest, file buffers are allocated in the guest kernel. The file buffers are also mapped in the QEMU process on the host, as well as in the virtiofsd process.

For example, if you use 300Mi of file buffer cache in the guest, both QEMU and virtiofsd appear to use 300Mi additional memory. However, the same memory is being used in all three cases. In other words, the total memory usage is only 300Mi, mapped in three different places. This is correctly accounted for when reporting the memory utilization metrics.

Before running OpenShift sandboxed containers, you can check whether the nodes in your cluster are eligible to run Kata containers. Some cluster nodes might not comply with sandboxed containers' minimum requirements. The most common reason for node ineligibility is the lack of virtualization support on the node. If you attempt to run sandboxed workloads on ineligible nodes, you will experience errors. You can use the Node Feature Discovery (NFD) Operator and a NodeFeatureDiscovery resource to automatically check node eligibility.

The following procedure applies the feature.node.kubernetes.io/runtime.kata=true label to all eligible nodes and configures the KataConfig resource to check for node eligibility.

You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console.

The OpenShift sandboxed containers Operator is now installed on your cluster.

You must create one KataConfig custom resource (CR) to enable installing kata as a RuntimeClass on your cluster nodes.

The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Wait for the kata installation to complete and the worker nodes to reboot before continuing to the next step.

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload in a sandboxed container, you must manually add kata as the runtimeClassName to the workload YAML file.

OpenShift Container Platform creates the workload and begins scheduling it.

You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI.

The OpenShift sandboxed containers Operator is now installed on your cluster.

You must create one KataConfig custom resource (CR) to install kata as a RuntimeClass on your nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following:

The new KataConfig CR is created and begins to install kata as a RuntimeClass on the worker nodes. Wait for the `kata`installation to complete and the worker nodes to reboot before continuing to the next step.

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload in a sandboxed container, you must add kata as the runtimeClassName to the workload YAML file.

apiVersion: v1
kind: Pod
metadata:
  name: hello-openshift
  labels:
    app: hello-openshift
spec:
  runtimeClassName: kata
  containers:
    - name: hello-openshift
      image: quay.io/openshift/origin-hello-openshift
      ports:
        - containerPort: 8888
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1001
        capabilities:
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault

OpenShift Container Platform creates the workload and begins scheduling it.

To uninstall OpenShift sandboxed containers, you must first delete all running pods that use kata as the runtimeClass.

Deleting the KataConfig custom resource (CR) removes and uninstalls the kata runtime and its related resources from your cluster.

Wait for the Kata runtime and resources to uninstall and for the worker nodes to reboot before continuing to the next step.

Deleting the OpenShift sandboxed containers Operator removes the catalog subscription, Operator group, and cluster service version (CSV) for that Operator.

After you run the preceding commands, your cluster is restored to the state that it was prior to the installation process. You can now revoke namespace access to the Operator by deleting the openshift-sandboxed-containers-operator namespace.

The KataConfig custom resource definition (CRD) lets you define the KataConfig CR. To complete the uninstall process, delete the KataConfig CRD from your cluster.

To uninstall OpenShift sandboxed containers, you must first delete all running pods that use kata as the runtimeClass.

Remove and uninstall the kata runtime and all its related resources, such as CRI-O config and RuntimeClass, from your cluster. The deletion typically takes between ten to forty minutes, depending on the size of the deployment.

The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

Remove the OpenShift sandboxed containers Operator from your cluster by deleting the Operator subscription, Operator group, cluster service version (CSV), and namespace.

The KataConfig custom resource definition (CRD) lets you define the KataConfig CR. Delete the KataConfig CRD from your cluster.

The KataConfig YAML file in the OpenShift Container Platform contains the version number for the monitor image. Update the version number with the correct version.

You can manually patch the monitor image in the KataConfig CR to update the monitor pods.

The oc adm must-gather CLI command collects the information from your cluster that is most likely needed for debugging issues, including:

By default, the oc adm must-gather command uses the default plugin image and writes into ./must-gather.local.

Alternatively, you can collect specific information by running the command with the appropriate arguments as described in the following sections:

When you run oc adm must-gather, a new pod with a random name is created in a new project on the cluster. The data is collected on that pod and saved in a new directory that starts with must-gather.local. This directory is created in the current working directory.

For example:

NAMESPACE                      NAME                 READY   STATUS      RESTARTS      AGE
...
openshift-must-gather-5drcj    must-gather-bklx4    2/2     Running     0             72s
openshift-must-gather-5drcj    must-gather-s8sdh    2/2     Running     0             72s
...

To collect OpenShift sandboxed containers data with must-gather, you must specify the OpenShift sandboxed containers image:

--image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel8:1.2.0

Cluster administrators can use the enhanced debug logs for OpenShift sandboxed containers to troubleshoot issues. The logs for each node are printed to the node journal.

You can review the logs for the following OpenShift sandboxed containers components:

Logs for QEMU do not print to the node journal. However, a QEMU failure is reported to the runtime, and the console of the QEMU guest is printed to the node journal. You can view these logs together with the Kata agent logs.