Chapter 2. Deploying confidential containers on bare metal

You can deploy confidential containers workloads on a Red Hat OpenShift Container Platform cluster running on bare metal.

Important

Confidential containers on bare metal is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

2.1. Preparation
Copy link

Review these prerequisites and concepts before you deploy confidential containers on bare metal.

2.1.1. Prerequisites
Copy link

You have installed the latest version of Red Hat OpenShift Container Platform on the cluster where you are running your confidential containers workload.
You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.

2.1.2. Initdata
Copy link

The initdata specification provides a flexible way to initialize a pod with workload-specific data at runtime, avoiding the need to embed such data in the virtual machine (VM) image.

This approach enhances security by reducing the exposure of confidential information and improves flexibility by eliminating custom image builds. For example, initdata can include three configuration settings:

An X.509 certificate for secure communication.
A cryptographic key for authentication.
An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

The initdata content configures the following components:

Attestation Agent (AA), which verifies the trustworthiness of the pod by sending evidence for attestation.
Confidential Data Hub (CDH), which manages secrets and secure data access within the pod VM.
Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

You create an initdata.toml file and convert it to a Base64-encoded, gzip-format string. You apply the initdata string to your workload by adding an annotation to the pod manifest.

2.1.3. Kata runtime deployment modes
Copy link

You can choose how the Operator installs and configures the Kata runtime using the deployment modes MachineConfig, DaemonSet, or DaemonSetFallback. You specify the data.deploymentMode key in the osc-feature-gates config map. This flexibility allows the Operator to work consistently in clusters with or without the Machine Config Operator (MCO).

MachineConfig: For clusters that use the Machine Config Operator (MCO). If the deploymentMode key is missing in the config map, the Operator defaults to the MachineConfig for backward compatibility.
DaemonSet: For clusters without the MCO. The Operator uses a DaemonSet to install kata-containers RPMs and manage CRI-O configuration by using host drop-in files. Installation progress is tracked through node labels (for example, installing, installed).
DaemonSetFallback: Enables conditional deployment based on the cluster environment. When set, the operator checks for the presence of the MCO. It uses DaemonSet if the MachineConfig add-on is unavailable and defaults to MachineConfig otherwise.

2.2. Deployment overview
Copy link

You deploy confidential containers on bare metal by performing the following steps:

Create MachineConfig for TDX.
Install the OpenShift sandboxed containers Operator.
Configure the TDX remote attestation infrastructure for TDX workloads.
Enable the confidential containers feature gate.
Create the KataConfig CR.
Verify the attestation process.

2.3. Creating MachineConfig config map for TDX
Copy link

If you use Intel Trust Domain Extensions (TDX), you must create a MachineConfig object before you install the Red Hat build of Trustee Operator.

Procedure

Create a tdx-machine-config.yaml manifest file according to the following example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: <role> 
  name: 99-enable-intel-tdx
spec:
  kernelArguments:
  - kvm_intel.tdx=1
  - nohibernate
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/modules-load.d/vsock.conf
          mode: 0644
          contents:
            source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: <role>


  name: 99-enable-intel-tdx
spec:
  kernelArguments:
  - kvm_intel.tdx=1
  - nohibernate
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/modules-load.d/vsock.conf
          mode: 0644
          contents:
            source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

Copy to Clipboard

Toggle word wrap

1: Specify master for single-node OpenShift or kata-oc for a multi-node cluster.

Create the TDX config map by running the following command:
```
oc create -f tdx-config.yaml
```
```
$ oc create -f tdx-config.yaml
```
Copy to Clipboard Toggle word wrap

2.4. Installing the OpenShift sandboxed containers Operator
Copy link

You install the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

Create an osc-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

Copy to Clipboard

Toggle word wrap

Create the namespace by running the following command:
```
oc apply -f osc-namespace.yaml
```
```
$ oc apply -f osc-namespace.yaml
```
Copy to Clipboard Toggle word wrap

Create an osc-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

Copy to Clipboard

Toggle word wrap

Create the operator group by running the following command:
```
oc apply -f osc-operatorgroup.yaml
```
```
$ oc apply -f osc-operatorgroup.yaml
```
Copy to Clipboard Toggle word wrap

Create an osc-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.0

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.0

Copy to Clipboard

Toggle word wrap

Create the subscription by running the following command:
```
oc create -f osc-subscription.yaml
```
```
$ oc create -f osc-subscription.yaml
```
Copy to Clipboard Toggle word wrap
Verify that the Operator is correctly installed by running the following command:
```
oc get csv -n openshift-sandboxed-containers-operator
```
```
$ oc get csv -n openshift-sandboxed-containers-operator
```
Copy to Clipboard Toggle word wrap
This command can take several minutes to complete.

Watch the process by running the following command:

watch oc get csv -n openshift-sandboxed-containers-operator

$ watch oc get csv -n openshift-sandboxed-containers-operator

Copy to Clipboard

Toggle word wrap

Example output

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.0    1.10.3        Succeeded

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.0    1.10.3        Succeeded

Copy to Clipboard

Toggle word wrap

2.5. Configuring auto-detection of TEEs
Copy link

You must configure your nodes so that the OpenShift sandboxed containers Operator can detect the Trusted Execution Environments (TEEs).

You label the nodes by installing and configuring the Node Feature Discovery (NFD) Operator.

2.5.1. Creating a NodeFeatureDiscovery custom resource
Copy link

You create a NodeFeatureDiscovery custom resource (CR) to define the configuration parameters that the Node Feature Discovery (NFD) Operator checks to automatically detect your TEE.

Prerequisites

You have installed the NFD Operator. For more information, see Node Feature Discovery Operator in the OpenShift Container Platform documentation.

Procedure

Create a my-nfd.yaml manifest file according to the following example:

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  operand:
    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
    imagePullPolicy: Always
    servicePort: 12000
  workerConfig:
    configData: |

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  operand:
    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
    imagePullPolicy: Always
    servicePort: 12000
  workerConfig:
    configData: |

Copy to Clipboard

Toggle word wrap

Create the NodeFeatureDiscovery CR:
```
oc create -f my-nfd.yaml
```
```
$ oc create -f my-nfd.yaml
```
Copy to Clipboard Toggle word wrap

2.5.2. Creating the NodeFeatureRule custom resource
Copy link

Create a NodeFeatureRule custom resource for your Trusted Execution Environment (TEE).

Procedure

Create a custom resource manifest named my-nodefeaturerule.yaml for your TEE:

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: osc-rules
  namespace: openshift-nfd
spec:
  rules:
    - name: "runtime.kata"
      labels:
        "feature.node.kubernetes.io/runtime.kata": "true"
      matchAny:
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: {op: Exists}
                VMX: {op: Exists}
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: {op: Exists}
                kvm_intel: {op: Exists}
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: {op: Exists}
                SVM: {op: Exists}
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: {op: Exists}
                kvm_amd: {op: Exists}

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: osc-rules
  namespace: openshift-nfd
spec:
  rules:
    - name: "runtime.kata"
      labels:
        "feature.node.kubernetes.io/runtime.kata": "true"
      matchAny:
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: {op: Exists}
                VMX: {op: Exists}
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: {op: Exists}
                kvm_intel: {op: Exists}
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: {op: Exists}
                SVM: {op: Exists}
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: {op: Exists}
                kvm_amd: {op: Exists}

Copy to Clipboard

Toggle word wrap

AMD SEV-SNP:

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: amd-sev-snp
  namespace: openshift-nfd
spec:
  rules:
    - name: "amd.sev-snp"
      labels:
        "amd.feature.node.kubernetes.io/snp": "true"
      extendedResources:
        sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
      matchFeatures:
        - feature: cpu.security
          matchExpressions:
            sev.snp.enabled: { op: Exists }

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: amd-sev-snp
  namespace: openshift-nfd
spec:
  rules:
    - name: "amd.sev-snp"
      labels:
        "amd.feature.node.kubernetes.io/snp": "true"
      extendedResources:
        sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
      matchFeatures:
        - feature: cpu.security
          matchExpressions:
            sev.snp.enabled: { op: Exists }

Copy to Clipboard

Toggle word wrap

Intel TDX:

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: intel-tdx
  namespace: openshift-nfd
spec:
  rules:
    - name: "intel.sgx"
      labels:
        "feature.node.kubernetes.io/sgx": "true"
      extendedResources:
        sgx.intel.com/epc: "@cpu.security.sgx.epc"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            SGX: {op: Exists}
            SGXLC: {op: Exists}
        - feature: cpu.security
          matchExpressions:
            sgx.enabled: {op: IsTrue}
        - feature: kernel.config
          matchExpressions:
            X86_SGX: {op: Exists}
    - name: "intel.tdx"
      labels:
        "intel.feature.node.kubernetes.io/tdx": "true"
      extendedResources:
        tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
      matchFeatures:
        - feature: cpu.security
          matchExpressions:
            tdx.enabled: {op: Exists}

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: intel-tdx
  namespace: openshift-nfd
spec:
  rules:
    - name: "intel.sgx"
      labels:
        "feature.node.kubernetes.io/sgx": "true"
      extendedResources:
        sgx.intel.com/epc: "@cpu.security.sgx.epc"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            SGX: {op: Exists}
            SGXLC: {op: Exists}
        - feature: cpu.security
          matchExpressions:
            sgx.enabled: {op: IsTrue}
        - feature: kernel.config
          matchExpressions:
            X86_SGX: {op: Exists}
    - name: "intel.tdx"
      labels:
        "intel.feature.node.kubernetes.io/tdx": "true"
      extendedResources:
        tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
      matchFeatures:
        - feature: cpu.security
          matchExpressions:
            tdx.enabled: {op: Exists}

Copy to Clipboard

Toggle word wrap

Create the NodeFeatureRule CR by running the following command:
```
oc create -f my-nodefeaturerule.yaml
```
```
$ oc create -f my-nodefeaturerule.yaml
```
Copy to Clipboard Toggle word wrap

Note

A relabeling delay of up to 1 minute might occur.

2.6. Configuring the TDX remote attestation infrastructure for TDX workloads
Copy link

You can deploy pod VM images for confidential containers bare-metal infrastructure, which includes the following components that provide remote attestation for Intel Trust Domain Extensions (TDX):

In-cluster Provisioning Certificate Caching Service (PCCS) deployment
Automatic per-node PCK Cert ID Retrieval Tool based platform (re-)registration
Per-node TDX-QGS quoting service

You can set up the namespaces, secrets, certificates, and environment variables required for your deployment.

Prerequisites

You have installed the Intel Device Plugins Operator and created an instance of the Intel Software Guard Extensions Device Plugin. For details, see Installing from the software catalog by using the web console in the OpenShift Container Platform documentation.
The node on which you deploy PCCS must have Internet access.

Procedure

Create the intel-dcap namespace by running the following command:
```
oc create namespace intel-dcap
```
```
$ oc create namespace intel-dcap
```
Copy to Clipboard Toggle word wrap
Switch to the intel-dcap project by running the following command:
```
oc project intel-dcap
```
```
$ oc project intel-dcap
```
Copy to Clipboard Toggle word wrap
Update the Security Context Constraint by running the following command:
```
oc adm policy add-scc-to-user privileged -z default
```
```
$ oc adm policy add-scc-to-user privileged -z default
```
Copy to Clipboard Toggle word wrap
Switch to the default project by running the following command:
```
oc project default
```
```
$ oc project default
```
Copy to Clipboard Toggle word wrap
Set the PCCS_API_KEY variable by running the following command:
```
export PCCS_API_KEY="${PCCS_API_KEY:-}"
```
```
$ export PCCS_API_KEY="${PCCS_API_KEY:-}"
```
Copy to Clipboard Toggle word wrap
PCCS_API_KEY
The API key for Intel® SGX and Intel® TDX Provisioning Certification Service. Navigate to Intel Trusted Services API portal, sign in, and subscribe to the Intel® SGX and Intel® TDX Provisioning Certification Service. The API key is displayed on the Manage Subscriptions page.
Set the PCCS_USER_TOKEN variable by running the following command:
```
export PCCS_USER_TOKEN="${PCCS_USER_TOKEN:-mytoken}"
```
```
$ export PCCS_USER_TOKEN="${PCCS_USER_TOKEN:-mytoken}"
```
Copy to Clipboard Toggle word wrap
PCCS_USER_TOKEN
Specify the PCCS user token. For details, see the Design Guide for Intel® SGX Provisioning Certificate Caching Service (Intel® SGX PCCS).
Set the PCCS_ADMIN_TOKEN variable by running the following command:
```
export PCCS_ADMIN_TOKEN="${PCCS_ADMIN_TOKEN:-mytoken}"
```
```
$ export PCCS_ADMIN_TOKEN="${PCCS_ADMIN_TOKEN:-mytoken}"
```
Copy to Clipboard Toggle word wrap
PCCS_ADMIN_TOKEN
Specify the PCCS administration token. For details, see the Design Guide for Intel® SGX Provisioning Certificate Caching Service (Intel® SGX PCCS).

Set the value for the PCCS_NODE variable to the name of the control plane node by running the following command:

export PCCS_NODE=$(oc get nodes \
  -l 'node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=' \
  -o jsonpath='{.items[0].metadata.name}')

$ export PCCS_NODE=$(oc get nodes \
  -l 'node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=' \
  -o jsonpath='{.items[0].metadata.name}')

Copy to Clipboard

Toggle word wrap

Set the cluster-wide proxy variable:

If you are using a cluster-wide proxy, run the following command:

export CLUSTER_HTTPS_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.httpsProxy})"

$ export CLUSTER_HTTPS_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.httpsProxy})"

Copy to Clipboard

Toggle word wrap

If you are not using a proxy, run the following command:

export CLUSTER_NO_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.noProxy})"

$ export CLUSTER_NO_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.noProxy})"

Copy to Clipboard

Toggle word wrap

Generate a SHA-512 hash of the PCCS user token by running the following command:

export PCCS_USER_TOKEN_HASH=$(echo -n "$PCCS_USER_TOKEN" | sha512sum | tr -d '[:space:]-')

$ export PCCS_USER_TOKEN_HASH=$(echo -n "$PCCS_USER_TOKEN" | sha512sum | tr -d '[:space:]-')

Copy to Clipboard

Toggle word wrap

Generate a SHA-512 hash of the PCCS admin token by running the following command:

export PCCS_ADMIN_TOKEN_HASH=$(echo -n "$PCCS_ADMIN_TOKEN" | sha512sum | tr -d '[:space:]-')

$ export PCCS_ADMIN_TOKEN_HASH=$(echo -n "$PCCS_ADMIN_TOKEN" | sha512sum | tr -d '[:space:]-')

Copy to Clipboard

Toggle word wrap

Create a temporary directory for the PCCS certificate and key by running the following command:
```
PCCS_PEM_CERT_PATH=$(mktemp -d)
```
```
$ PCCS_PEM_CERT_PATH=$(mktemp -d)
```
Copy to Clipboard Toggle word wrap

Generate a self-signed PCCS certificate and private key by running the following command:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout $PCCS_PEM_CERT_PATH/private.pem \
  -out $PCCS_PEM_CERT_PATH/certificate.pem \
  -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout $PCCS_PEM_CERT_PATH/private.pem \
  -out $PCCS_PEM_CERT_PATH/certificate.pem \
  -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

Copy to Clipboard

Toggle word wrap

Set the value of the PCCS_PEM to the Base64-encoded PCCS private key by running the following command:

export PCCS_PEM=$(cat "$PCCS_PEM_CERT_PATH"/private.pem | base64 | tr -d '\n')

$ export PCCS_PEM=$(cat "$PCCS_PEM_CERT_PATH"/private.pem | base64 | tr -d '\n')

Copy to Clipboard

Toggle word wrap

Set the value of PCCS_CERT to the Base64-encoded PCCS certificate by running the following command:

export PCCS_CERT=$(cat "$PCCS_PEM_CERT_PATH"/certificate.pem | base64 | tr -d '\n')

$ export PCCS_CERT=$(cat "$PCCS_PEM_CERT_PATH"/certificate.pem | base64 | tr -d '\n')

Copy to Clipboard

Toggle word wrap

Create a generic secret for PCCS by running the following command:

oc create secret generic pccs-secrets \
    --namespace intel-dcap \
    --from-literal=PCCS_API_KEY="$PCCS_API_KEY" \
    --from-literal=PCCS_USER_TOKEN_HASH="$PCCS_USER_TOKEN_HASH" \
    --from-literal=USER_TOKEN="$PCCS_USER_TOKEN" \
    --from-literal=PCCS_ADMIN_TOKEN_HASH="$PCCS_ADMIN_TOKEN_HASH"

$ oc create secret generic pccs-secrets \
    --namespace intel-dcap \
    --from-literal=PCCS_API_KEY="$PCCS_API_KEY" \
    --from-literal=PCCS_USER_TOKEN_HASH="$PCCS_USER_TOKEN_HASH" \
    --from-literal=USER_TOKEN="$PCCS_USER_TOKEN" \
    --from-literal=PCCS_ADMIN_TOKEN_HASH="$PCCS_ADMIN_TOKEN_HASH"

Copy to Clipboard

Toggle word wrap

2.7. Creating the osc-feature-gates config map
Copy link

You enable the confidential containers feature gate and specify the deployment mode by creating the config map.

Procedure

Create a my-feature-gate.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: <deployment_mode>
```
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: <deployment_mode>
```
Copy to Clipboard Toggle word wrap
where
<deployment_mode>
On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:
MachineConfig for clusters that always use the MCO
DaemonSet for clusters that never use the MCO
DaemonSetFallback for clusters that sometimes use the MCO
Create the my-feature-gates config map by running the following command:
```
oc create -f my-feature-gate.yaml
```
```
$ oc create -f my-feature-gate.yaml
```
Copy to Clipboard Toggle word wrap

2.8. Creating initdata
Copy link

You create initdata to securely initialize a pod with sensitive or workload-specific data at runtime, thus avoiding the need to embed this data in a virtual machine image. This approach provides additional security by reducing the risk of exposure of confidential information and eliminates the need for custom image builds.

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

Procedure

Create the initdata.toml file:

algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''

"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''

"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

Copy to Clipboard

Toggle word wrap

url: Specify the Red Hat build of Trustee
<kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.
image_security_policy_uri: Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

Convert the initdata.toml file to a Base64-encoded string in gzip format in a text file by running the following command:
```
cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
```
$ cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
Copy to Clipboard Toggle word wrap
Record this string to use in the pod manifest.
Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:
```
hash=$(sha256sum initdata.toml | cut -d' ' -f1)
```
```
$ hash=$(sha256sum initdata.toml | cut -d' ' -f1)
```
Copy to Clipboard Toggle word wrap

Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

Copy to Clipboard

Toggle word wrap

Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:
```
PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
```
```
$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
```
Copy to Clipboard Toggle word wrap
Record the PCR8_HASH value for the RVPS config map.

2.9. Applying initdata to a pod
Copy link

Prerequisite

You have created an initdata string.

Procedure

Add the initdata string to the pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-cc
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-cc
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

Copy to Clipboard

Toggle word wrap

Create the pod by running the following command:
```
oc create -f my-pod.yaml
```
```
$ oc create -f my-pod.yaml
```
Copy to Clipboard Toggle word wrap

2.10. Creating the KataConfig custom resource
Copy link

You must create the KataConfig custom resource (CR) to install kata-cc as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-cc as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

A large OpenShift Container Platform deployment with a greater number of worker nodes.
Activation of the BIOS and Diagnostics utility.
Deployment on a hard disk drive rather than an SSD.
Deployment on physical nodes such as bare metal, rather than on virtual nodes.
A slow CPU and network.

Procedure

Create an example-kataconfig.yaml manifest file according to the following example:

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:

  enablePeerPods: false
  checkNodeEligibility: true

  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:

  enablePeerPods: false
  checkNodeEligibility: true

  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'

Copy to Clipboard

Toggle word wrap

Create the KataConfig CR by running the following command:
```
oc create -f example-kataconfig.yaml
```
```
$ oc create -f example-kataconfig.yaml
```
Copy to Clipboard Toggle word wrap
The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.
Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.
Monitor the installation progress by running the following command:
```
watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
```
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
Copy to Clipboard Toggle word wrap
When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.
Verify the runtime classes by running the following command:
```
oc get runtimeclass
```
```
$ oc get runtimeclass
```
Copy to Clipboard Toggle word wrap
Example output
```
NAME             HANDLER          AGE
kata-cc      kata-tdx      152m
```
```
NAME             HANDLER          AGE
kata-cc      kata-tdx      152m
```
Copy to Clipboard Toggle word wrap

2.11. Verifying attestation
Copy link

You can verify the attestation process by creating a test pod to retrieve a specific resource from Red Hat build of Trustee.

Important

This procedure is an example to verify that attestation is working. Do not write sensitive data to standard I/O, because the data can be captured by using a memory dump. Only data written to memory is encrypted.

Procedure

Create a test-pod.yaml manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 
spec:
  runtimeClassName: kata-cc
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
metadata:
  name: coco-test-pod
  labels:
    app: coco-test-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 
spec:
  runtimeClassName: kata-cc
  containers:
    - name: test-container
      image: registry.access.redhat.com/ubi9/ubi:9.3
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>


spec:
  runtimeClassName: kata-cc
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
metadata:
  name: coco-test-pod
  labels:
    app: coco-test-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>


spec:
  runtimeClassName: kata-cc
  containers:
    - name: test-container
      image: registry.access.redhat.com/ubi9/ubi:9.3
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

Copy to Clipboard

Toggle word wrap

1 1 2: Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

Create the pod by running the following command:
```
oc create -f test-pod.yaml
```
```
$ oc create -f test-pod.yaml
```
Copy to Clipboard Toggle word wrap
Log in to the pod by running the following command:
```
oc exec -it ocp-cc-pod -- bash
```
```
$ oc exec -it ocp-cc-pod -- bash
```
Copy to Clipboard Toggle word wrap
Fetch the Red Hat build of Trustee resource by running the following command:
```
curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status
```
```
$ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status
```
Copy to Clipboard Toggle word wrap
Example output
```
success #/
```
```
success #/
```
Copy to Clipboard Toggle word wrap

Chapter 2. Deploying confidential containers on bare metal

2.1. Preparation
Copy link

2.1.1. Prerequisites
Copy link

2.1.2. Initdata
Copy link

2.1.3. Kata runtime deployment modes
Copy link

2.2. Deployment overview
Copy link

2.3. Creating MachineConfig config map for TDX
Copy link

2.4. Installing the OpenShift sandboxed containers Operator
Copy link

2.5. Configuring auto-detection of TEEs
Copy link

2.5.1. Creating a NodeFeatureDiscovery custom resource
Copy link

2.5.2. Creating the NodeFeatureRule custom resource
Copy link

2.6. Configuring the TDX remote attestation infrastructure for TDX workloads
Copy link

2.7. Creating the osc-feature-gates config map
Copy link

2.8. Creating initdata
Copy link

2.9. Applying initdata to a pod
Copy link

2.10. Creating the KataConfig custom resource
Copy link

2.11. Verifying attestation
Copy link

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 2. Deploying confidential containers on bare metal

2.1. PreparationCopy linkLink copied to clipboard!

2.1.1. PrerequisitesCopy linkLink copied to clipboard!

2.1.2. InitdataCopy linkLink copied to clipboard!

2.1.3. Kata runtime deployment modesCopy linkLink copied to clipboard!

2.2. Deployment overviewCopy linkLink copied to clipboard!

2.3. Creating MachineConfig config map for TDXCopy linkLink copied to clipboard!

2.4. Installing the OpenShift sandboxed containers OperatorCopy linkLink copied to clipboard!

2.5. Configuring auto-detection of TEEsCopy linkLink copied to clipboard!

2.5.1. Creating a NodeFeatureDiscovery custom resourceCopy linkLink copied to clipboard!

2.5.2. Creating the NodeFeatureRule custom resourceCopy linkLink copied to clipboard!

2.6. Configuring the TDX remote attestation infrastructure for TDX workloadsCopy linkLink copied to clipboard!

2.7. Creating the osc-feature-gates config mapCopy linkLink copied to clipboard!

2.8. Creating initdataCopy linkLink copied to clipboard!

2.9. Applying initdata to a podCopy linkLink copied to clipboard!

2.10. Creating the KataConfig custom resourceCopy linkLink copied to clipboard!

2.11. Verifying attestationCopy linkLink copied to clipboard!

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

2.1. Preparation
Copy link

2.1.1. Prerequisites
Copy link

2.1.2. Initdata
Copy link

2.1.3. Kata runtime deployment modes
Copy link

2.2. Deployment overview
Copy link

2.3. Creating MachineConfig config map for TDX
Copy link

2.4. Installing the OpenShift sandboxed containers Operator
Copy link

2.5. Configuring auto-detection of TEEs
Copy link

2.5.1. Creating a NodeFeatureDiscovery custom resource
Copy link

2.5.2. Creating the NodeFeatureRule custom resource
Copy link

2.6. Configuring the TDX remote attestation infrastructure for TDX workloads
Copy link

2.7. Creating the osc-feature-gates config map
Copy link

2.8. Creating initdata
Copy link

2.9. Applying initdata to a pod
Copy link

2.10. Creating the KataConfig custom resource
Copy link

2.11. Verifying attestation
Copy link