Chapter 5. Deploying confidential containers on IBM Z and IBM LinuxONE bare-metal servers

1. Install the OpenShift sandboxed containers Operator.
2. Configure auto-detection of TEEs.
3. Enable the confidential containers feature gate.
4. Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.
5. Upload a Secure Execution image to the container registry.
6. Create the kata-addon-artifacts config map.

7. Create initdata to initialize a pod with sensitive or workload-specific data at runtime.

   Important

   Do not use the default permissive Kata Agent policy in a production environment. You must configure a restrictive policy, preferably by creating initdata.

   As a minimum requirement, you must disable ExecProcessRequest to prevent a cluster administrator from accessing sensitive data by running the oc exec command on a confidential containers pod.

8. Apply initdata to a pod.
9. Create the KataConfig CR.
10. Verify the attestation process.

Procedure

1. Create a tdx-machine-config.yaml manifest file according to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: <role> 

   1

   
     name: 99-enable-intel-tdx
   spec:
     kernelArguments:
     - kvm_intel.tdx=1
     - nohibernate
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modules-load.d/vsock.conf
             mode: 0644
             contents:
               source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

   Copy to Clipboard Toggle word wrap

   1

   Specify master for single-node OpenShift or kata-oc for a multi-node cluster.

2. Create the TDX config map by running the following command:

   $ oc create -f tdx-config.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Manual
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.0

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Get the InstallPlan CR for the OpenShift sandboxed containers Operator by running the following command:

   $ oc get installplan -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   • Installation example output

     NAME            CSV                                      APPROVAL  APPROVED
     install-bl4fl   sandboxed-containers-operator.v1.11.0    Manual    false

     Copy to Clipboard Toggle word wrap

   • Upgrade example output

     NAME            CSV                                     APPROVAL   APPROVED
     install-jdzrb   sandboxed-containers-operator.v1.11.0   Manual     false
     install-pfk8l   sandboxed-containers-operator.v1.10.3   Manual     true

     Copy to Clipboard Toggle word wrap

8. Approve the manual installation by running the following command:

   $ oc patch installplan <installplan_name> -p '{"spec":{"approved":true}}' --type=merge -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   <installplan_name>

   Specify the InstallPlan resource. For example, install-jdzrb.

9. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

10. Watch the process by running the following command:

    $ watch oc get csv -n openshift-sandboxed-containers-operator

    Copy to Clipboard Toggle word wrap

    Example output

    NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.0    1.10.3        Succeeded

    Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"
     deploymentMode: daemonset

   Copy to Clipboard Toggle word wrap

   where

   <deployment_mode>

   On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:

   • MachineConfig for clusters that always use the MCO
   • DaemonSet for clusters that never use the MCO
   • DaemonSetFallback for clusters that sometimes use the MCO

2. Create the my-feature-gates config map by running the following command:

   $ oc create -f my-feature-gate.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Build a Secure Execution image.

2. Create a Dockerfile file for the Secure Execution image:

   FROM alpine:3.20
   RUN mkdir -p /images
   COPY ./<image_name> /images/<image_name>
   RUN chmod 644 /images/<image_name>

   Copy to Clipboard Toggle word wrap

   <image_name>

   Specify the custom Secure Execution image name. For example, se.img.

3. Build a container image with a custom tag from the Dockerfile:

   $ docker build -t <registry_name>/<user_name>/kata-se-artifacts:<image_tag> .

   Copy to Clipboard Toggle word wrap

4. Push the container image to your registry:

   $ docker push <registry_name>/<user_name>/kata-se-artifacts:<image_tag>

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the kata-addon-artifacts.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kata-addon-artifacts
     namespace: openshift-sandboxed-containers-operator
   data:
     addonImage: "<container_image_path>"
     kernelPath: "<kernel_path>"

   Copy to Clipboard Toggle word wrap

   <container_image_path>

   Specify the path to your container image in the registry. For example, quay.io/openshift_sandboxed_containers/kata-se-artifacts:v1.0.

   <kernel_path>

   Specify the kernel path inside the image. For example, /images/se.img.

2. Create the kata-addon-artifacts config map by running the following command:

   $ oc create -f kata-addon-artifacts.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the Red Hat build of Trustee IP address by running the following command:

   $ oc get node $(oc get pod -n trustee-operator-system \
     -o jsonpath='{.items[0].spec.nodeName}') \
     -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

   Copy to Clipboard Toggle word wrap

   Example output

   192.168.122.22

   Copy to Clipboard Toggle word wrap

2. Obtain the port by running the following command:

   $ oc get svc kbs-service -n trustee-operator-system

   Copy to Clipboard Toggle word wrap

   Example output

   NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
   kbs-service  NodePort    172.30.116.11   <none>        8080:32178/TCP   12d

   Copy to Clipboard Toggle word wrap

3. Create the initdata.toml file:

   algorithm = "sha384"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   '''
   
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   [image]
   image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
   '''

   Copy to Clipboard Toggle word wrap

   url

   Specify the Red Hat build of Trustee

   <kbs_certificate>

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

   image_security_policy_uri

   Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

4. Convert the initdata.toml file to a Base64-encoded string in gzip format in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Copy to Clipboard Toggle word wrap

   Record this string to use in the pod manifest.

Procedure

1. Add the initdata string to the pod manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-cc
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
   
     enablePeerPods: false
     checkNodeEligibility: true
   
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-cc on specific nodes, specify the key and value, for example, cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.

   Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.

4. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-cc      kata-se      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 

   1

   
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:latest
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault
   metadata:
     name: coco-test-pod
     labels:
       app: coco-test-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 

   2

   
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: test-container
         image: registry.access.redhat.com/ubi9/ubi:9.3
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

   1 2

   Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

   Copy to Clipboard Toggle word wrap

3. Log in to the pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

   Copy to Clipboard Toggle word wrap

4. Fetch the Red Hat build of Trustee resource by running the following command:

   $ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status

   Copy to Clipboard Toggle word wrap

   Example output

   success #/

   Copy to Clipboard Toggle word wrap