Red Hat SummitChapter 1. Discover
You can deploy Red Hat OpenShift sandboxed containers workloads on a Red Hat OpenShift Container Platform cluster running on bare-metal servers. OpenShift sandboxed containers integrates Kata containers as an optional runtime, providing enhanced security and isolation by running containerized applications in lightweight virtual machines.
This integration provides a more secure runtime environment for sensitive workloads without significant changes to existing OpenShift Container Platform workflows. This runtime supports containers in dedicated virtual machines (VMs), providing improved workload isolation.
1.1. Features
OpenShift sandboxed containers provides the following features:
- Run privileged or untrusted workloads
You can safely run workloads that require specific privileges, without the risk of compromising cluster nodes by running privileged containers. Workloads that require special privileges include the following:
- Workloads that require special capabilities from the kernel, beyond the default ones granted by standard container runtimes such as CRI-O, for example to access low-level networking features.
- Workloads that need elevated root privileges, for example to access a specific physical device. With OpenShift sandboxed containers, it is possible to pass only a specific device through to the virtual machines (VM), ensuring that the workload cannot access or misconfigure the rest of the system.
Workloads for installing or using
set-uidroot binaries. These binaries grant special privileges and, as such, can present a security risk. With OpenShift sandboxed containers, additional privileges are restricted to the virtual machines, and grant no special access to the cluster nodes.Some workloads require privileges specifically for configuring the cluster nodes. Such workloads should still use privileged containers, because running on a virtual machine would prevent them from functioning.
- Ensure isolation for sensitive workloads
- The OpenShift sandboxed containers for Red Hat OpenShift Container Platform integrates Kata containers as an optional runtime, providing enhanced security and isolation by running containerized applications in lightweight virtual machines. This integration provides a more secure runtime environment for sensitive workloads without significant changes to existing OpenShift workflows. This runtime supports containers in dedicated virtual machines (VMs), providing improved workload isolation.
- Ensure kernel isolation for each workload
-
You can run workloads that require custom kernel tuning (such as
sysctl, scheduler changes, or cache tuning) and the creation of custom kernel modules (such asout of treeor special arguments). - Share the same workload across tenants
-
You can run workloads that support many users (tenants) from different organizations sharing the same OpenShift Container Platform cluster. The system also supports running third-party workloads from multiple vendors, such as container network functions (CNFs) and enterprise applications. Third-party CNFs, for example, may not want their custom settings interfering with packet tuning or with
sysctlvariables set by other applications. Running inside a completely isolated kernel is helpful in preventing "noisy neighbor" configuration problems. - Ensure proper isolation and sandboxing for testing software
-
You can run containerized workloads with known vulnerabilities or handle issues in an existing application. This isolation enables administrators to give developers administrative control over pods, which is useful when the developer wants to test or validate configurations beyond those an administrator would typically grant. Administrators can, for example, safely and securely delegate kernel packet filtering (eBPF) to developers. eBPF requires
CAP_ADMINorCAP_BPFprivileges, and is therefore not allowed under a standard CRI-O configuration, as this would grant access to every process on the Container Host worker node. Similarly, administrators can grant access to intrusive tools such asSystemTap, or support the loading of custom kernel modules during their development. - Ensure default resource containment through VM boundaries
- By default, OpenShift sandboxed containers manages resources such as CPU, memory, storage, and networking in a robust and secure way. Since OpenShift sandboxed containers deploys on VMs, additional layers of isolation and security give a finer-grained access control to the resource. For example, an errant container will not be able to assign more memory than is available to the VM. Conversely, a container that needs dedicated access to a network card or to a disk can take complete control over that device without getting any access to other devices.
1.2. Compatibility with OpenShift Container Platform
You must ensure that your Red Hat OpenShift Container Platform version supports the features you require.
The required functionality for OpenShift Container Platform is supported by two main components:
- Kata runtime
- The Kata runtime is included with Red Hat Enterprise Linux CoreOS (RHCOS) and receives updates with every OpenShift Container Platform release. When enabling peer pods with the Kata runtime, the OpenShift sandboxed containers Operator requires external network connectivity to pull the necessary image components and helper utilities to create the pod virtual machine (VM) image.
- OpenShift sandboxed containers Operator
- The OpenShift sandboxed containers Operator is a Rolling Stream Operator, which means the latest version is the only supported version. It works with all currently supported versions of OpenShift Container Platform.
The Operator depends on the features that come with the RHCOS host and the environment it runs in.
You must install RHCOS on the worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported.
The following compatibility matrix for OpenShift sandboxed containers and OpenShift Container Platform releases identifies compatible features and environments.
| Architecture | OpenShift Container Platform version (without GPU) | OpenShift Container Platform version (with GPU) |
|---|---|---|
| x86_64 | 4.18.38+ | 4.21.9+ |
| s390x | 4.18.38+ | — |
There are two ways to deploy the Kata containers runtime:
- Bare metal
- Peer pods
You can deploy OpenShift sandboxed containers by using peer pods on Microsoft Azure, Amazon Web Services (AWS), or Google Cloud. With the release of OpenShift sandboxed containers 1.12.0, the OpenShift sandboxed containers Operator requires OpenShift Container Platform version 4.18.38 or later for deployments without GPU support .
The following table describes OpenShift Container Platform versions and features with the following support levels:
- GA: General Availability
- TP: Technology Preview
- DP: Developer Preview
The version numbers in the table represent the minimum supported version. For example, "4.21.9+" means version 4.21.9 or any later version.
| Platform | GPU | 4.18.38+ | 4.19.28+ | 4.20.18+ | 4.21.9+ |
|---|---|---|---|---|---|
| Bare metal | No | GA | GA | GA | GA |
| NVIDIA H100 | — | — | — | TP | |
| IBM Z bare metal | No | GA | GA | GA | GA |
| IBM Z peer pods | No | GA | GA | GA | GA |
| Azure | No | GA | GA | GA | GA |
| NVIDIA H100 | — | — | — | DP | |
| AWS | No | GA | GA | GA | GA |
| NVIDIA H100 | — | — | — | — | |
| Google Cloud | No | GA | GA | GA | GA |
| NVIDIA H100 | — | — | — | — |
1.3. Common terms
The following terms are used throughout the documentation.
- Attestation
- The process of verifying the integrity and trustworthiness of a Trusted Execution Environment (TEE) and the confidential containers workloads running within it, ensuring that only trusted code and data are executed. Red Hat build of Trustee performs this function.
- Confidential containers
- A technology that provides a confidential computing environment to protect containers and data by leveraging Trusted Execution Environments.
- Initdata
- A specification used to securely initialize a pod with workload-specific data (such as certificates, cryptographic keys, or an optional Kata Agent policy) at runtime, preventing the need to embed this data directly in the virtual machine (VM) image.
- Kata Agent
- A component within the pod Virtual Machine (VM) that enforces runtime policies and manages the lifecycle of the containers running inside the VM. Its policy controls API requests for peer pods.
- Kata containers
- Kata containers is a core upstream project that is used to build OpenShift sandboxed containers. OpenShift sandboxed containers integrate Kata containers with OpenShift Container Platform.
kataruntime- The optional runtime installed by the OpenShift sandboxed containers Operator when configuring bare metal deployments.
kata-ccruntime- The runtime class used specifically for confidential containers deployments on bare-metal servers.
kata-remoteruntime- The runtime class used for peer pod deployments on cloud platforms or remote hypervisors.
KataConfig- A custom resource used to configure and launch OpenShift sandboxed containers.
TrusteeConfig- A custom resource used to configure and launch Red Hat build of Trustee.
- OpenShift sandboxed containers
- OpenShift sandboxed containers integrates Kata containers as an optional runtime to provide enhanced security and isolation for container workloads by running applications in lightweight virtual machines.
- OpenShift sandboxed containers Operator
- The OpenShift sandboxed containers Operator manages the lifecycle of OpenShift sandboxed containers and confidential containers on a cluster.
- Peer pod
A peer pod in OpenShift sandboxed containers extends the concept of a standard pod. Unlike a standard sandboxed container, where the virtual machine is created on the worker node itself, in a peer pod, the virtual machine is created through a remote hypervisor using any supported hypervisor or cloud provider API.
The peer pod acts as a regular pod on the worker node, with its corresponding VM running elsewhere. The remote location of the VM is transparent to the user and is specified by the runtime class in the pod specification. The peer pod design circumvents the need for nested virtualization.
- Pod
A pod is a construct that is inherited from Kubernetes and OpenShift Container Platform. It represents resources where containers can be deployed. Containers run inside pods, and pods are used to specify resources that can be shared between multiple containers.
In the context of OpenShift sandboxed containers, a pod is implemented as a virtual machine. Several containers can run in the same pod on the same virtual machine.
- Red Hat build of Trustee
- Red Hat build of Trustee is an attestation service that verifies the trustworthiness of the location where you plan to run your workload or where you plan to send confidential information. Red Hat build of Trustee includes components deployed on a trusted side and used to verify whether the remote workload is running in a Trusted Execution Environment (TEE).
- Red Hat build of Trustee Operator
- The Red Hat build of Trustee Operator manages the installation, lifecycle, and configuration of Red Hat build of Trustee.
- Runtime class
- An object that describes the specific runtime configuration used to execute a workload.
- Sandbox
A sandbox is an isolated environment where programs can run. In a sandbox, you can run untested or untrusted programs without risking harm to the host machine or the operating system.
In the context of OpenShift sandboxed containers, sandboxing is achieved by running workloads in a different kernel using virtualization, providing enhanced control over the interactions between multiple workloads that run on the same host.
- Trusted Execution Environment (TEE)
- Hardware-based security technology leveraged by confidential containers to protect containers and data. Examples: Intel® TDX, AMD SEV-SNP.
1.4. OpenShift sandboxed containers Operator
The OpenShift sandboxed containers Operator encapsulates all of the components from Kata containers. It manages installation, lifecycle, and configuration tasks.
The OpenShift sandboxed containers Operator is packaged in the Operator bundle format as two container images:
- The bundle image contains metadata and is required to make the operator OLM-ready.
-
The second container image contains the actual controller that monitors and manages the
KataConfigresource.
The OpenShift sandboxed containers Operator is based on the Red Hat Enterprise Linux CoreOS (RHCOS) extensions concept. RHCOS extensions are a mechanism to install optional OpenShift Container Platform software. The OpenShift sandboxed containers Operator uses this mechanism to deploy sandboxed containers on a cluster.
The sandboxed containers RHCOS extension contains RPMs for Kata, QEMU, and its dependencies. You can enable them by using the MachineConfig resources that the Machine Config Operator provides.
Additional resources
1.5. OpenShift Virtualization
You can deploy OpenShift sandboxed containers on clusters with OpenShift Virtualization.
To run OpenShift Virtualization and OpenShift sandboxed containers at the same time, your virtual machines must be live migratable so that they do not block node reboots.
1.6. FIPS compliance
OpenShift Container Platform is designed for Federal Information Processing Standards (FIPS) 140-2 and 140-3. When running Red Hat Enterprise Linux (RHEL) or Red Hat Enterprise Linux CoreOS (RHCOS) booted in FIPS mode, OpenShift Container Platform core components use the RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le, and s390x architectures.
For more information about the NIST validation program, see Cryptographic Module Validation Program. For the latest NIST status for the individual versions of RHEL cryptographic libraries that have been submitted for validation, see Compliance Activities and Government Standards.
OpenShift sandboxed containers can be used on FIPS enabled clusters.
When running in FIPS mode, OpenShift sandboxed containers components, VMs, and VM images are adapted to comply with FIPS.
FIPS compliance for OpenShift sandboxed containers only applies to the kata runtime class. The peer pod runtime class, kata-remote, is not yet fully supported and has not been tested for FIPS compliance.
FIPS compliance is one of the most critical components required in highly secure environments, to ensure that only supported cryptographic technologies are allowed on nodes.
The use of FIPS Validated / Modules in Process cryptographic libraries is only supported on OpenShift Container Platform deployments on the x86_64 architecture.
To understand Red Hat’s view of OpenShift Container Platform compliance frameworks, refer to the Risk Management and Regulatory Readiness chapter of the OpenShift Security Guide Book.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}