Deploying confidential containers on bare-metal servers

Procedure

1. Create a tdx-machine-config.yaml manifest file according to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: <role>
     name: 99-enable-intel-tdx
   spec:
     kernelArguments:
     - kvm_intel.tdx=1
     - nohibernate
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modules-load.d/vsock.conf
             mode: 0644
             contents:
               source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

   <role>

   Specify master for single-node OpenShift or kata-oc for a multi-node cluster.

2. Create the config map by running the following command:

   $ oc create -f tdx-machine-config.yaml

   Updating the machine config triggers node reboot.

Verification

1. Verify that the machine config is correctly configured by running the following command:

   $ oc get machineconfig 99-enable-intel-tdx

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.12.0

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION         PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.12.0          Succeeded

Procedure

1. Configure the remote attestation project:

   1. Create the intel-dcap namespace by running the following command:

      $ oc create namespace intel-dcap

   2. Switch to the intel-dcap project by running the following command:

      $ oc project intel-dcap

   3. Create dedicated service accounts for PCCS and QGS by running the following commands:

      $ oc create serviceaccount pccs-sa -n intel-dcap

      $ oc create serviceaccount qgs-sa -n intel-dcap

   4. Grant the privileged Security Context Constraint to the service accounts by running the following commands:

      $ oc adm policy add-scc-to-user privileged -z pccs-sa -n intel-dcap

      $ oc adm policy add-scc-to-user privileged -z qgs-sa -n intel-dcap

2. Switch to the default project by running the following command:

   $ oc project default

3. Set the PCCS variables by running the following commands:

   $ export PCCS_API_KEY="<API_KEY_VALUE>"

   To obtain the API key for the Intel® Software Guard Extensions and Intel® TDX Provisioning Certification Service, navigate to the Intel Trusted Services API portal, sign in, and subscribe to the Provisioning Certification Service. The API key is displayed on the Manage Subscriptions page.

   $ export PCCS_USER_TOKEN="${PCCS_USER_TOKEN:-mytoken}"

   For details about PCCS tokens, see the Design Guide for Intel® SGX Provisioning Certificate Caching Service (Intel® SGX PCCS).

   $ export PCCS_ADMIN_TOKEN="${PCCS_ADMIN_TOKEN:-mytoken}"

   $ export PCCS_NODE=$(oc get nodes \
     -l 'node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=' \
     -o jsonpath='{.items[0].metadata.name}')

4. Set the cluster proxy variable by running the appropriate command:

   $ export CLUSTER_HTTPS_PROXY="$(oc get proxy/cluster \
     -o jsonpath={.spec.httpsProxy})"

   $ export CLUSTER_NO_PROXY="$(oc get proxy/cluster \
     -o jsonpath={.spec.noProxy})"

5. Create the PCCS secrets:

   1. Set the PCCS secrets variables by running the following commands:

      $ export PCCS_USER_TOKEN_HASH=$(echo -n "$PCCS_USER_TOKEN" | sha512sum | tr -d '[:space:]-')

      $ export PCCS_ADMIN_TOKEN_HASH=$(echo -n "$PCCS_ADMIN_TOKEN" | sha512sum | tr -d '[:space:]-')

      $ export PCCS_PEM_CERT_PATH=$(mktemp -d)

      Note

      This directory is automatically deleted at reboot. To re-use the PCCS certificate and key, you must create a persistent directory.

   2. Generate an RSA key pair and output the private key as a PCCS certificate by running the following command:

      $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
        -keyout $PCCS_PEM_CERT_PATH/private.pem \
        -out $PCCS_PEM_CERT_PATH/certificate.pem \
        -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

   3. Set the PCCS certificate variables by running the following commands:

      $ export PCCS_PEM=$(cat "$PCCS_PEM_CERT_PATH"/private.pem | base64 | tr -d '\n')

      $ export PCCS_CERT=$(cat "$PCCS_PEM_CERT_PATH"/certificate.pem | base64 | tr -d '\n')

   4. Create the PCCS secrets by running the following command:

      $ oc create secret generic pccs-secrets \
          --namespace intel-dcap \
          --from-literal=PCCS_API_KEY="$PCCS_API_KEY" \
          --from-literal=PCCS_USER_TOKEN_HASH="$PCCS_USER_TOKEN_HASH" \
          --from-literal=USER_TOKEN="$PCCS_USER_TOKEN" \
          --from-literal=PCCS_ADMIN_TOKEN_HASH="$PCCS_ADMIN_TOKEN_HASH"

6. Create the PCCS by running the following command:

   $ oc apply -f <(curl -sSf https://github.com/openshift/sandboxed-containers-operator/blob/v{osc-version-z}/scripts/install-helpers/baremetal-coco/intel-dcap/pccs.yaml.in|envsubst)

7. Configure the PCCS deployment to use the dedicated service account by running the following command:

   $ oc set serviceaccount deployment/pccs pccs-sa -n intel-dcap

8. Create the QGS by running the following command:

   $ oc apply -f https://github.com/openshift/sandboxed-containers-operator/blob/v{osc-version-z}/scripts/install-helpers/baremetal-coco/intel-dcap/qgs.yaml

9. Configure the QGS DaemonSet to use the dedicated service account by running the following command:

   $ oc set serviceaccount daemonset/tdx-qgs qgs-sa -n intel-dcap

Procedure

1. Create a my-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"
     deploymentMode: <deployment_mode>

   <deployment_mode>

   Specify the deployment mode.

2. Create the config map by running the following command:

   $ oc create -f my-feature-gate.yaml

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: false
     checkNodeEligibility: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-cc on specific nodes, specify the key and value, for example, kata-cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.

   Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.

4. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata-cc    kata-tdx        152m

   You can also see the default kata runtime class in addition to kata-cc.

Procedure

1. Obtain the Red Hat build of Trustee URL by running the following command:

   $ TRUSTEE_URL=$(oc get route kbs-service \
     -n trustee-operator-system -o jsonpath='{.spec.host}') \
     && echo $TRUSTEE_URL

2. Create the initdata.toml file:

   algorithm = <algorithm>
   version = "0.1.0"
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   '''
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   [image]
   image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
   '''
   
   "policy.rego" = '''
   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   default ExecProcessRequest := false
   '''

   algorithm

   Specify sha256, sha384, or sha512.

   url

   Specify Red Hat build of Trustee

   <kbs_certificate>

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

   image_security_policy_uri

   Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

3. Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Record this string to use in the pod manifest.

4. Calculate the hash of an initdata.toml file and assign its value to the hash variable by running the following command:

   $ hash=$(<algorithm> initdata.toml | cut -d' ' -f1)

5. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

6. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Record the PCR8_HASH value for the RVPS config map.

Procedure

1. Add the initdata string to the pod manifest and save the file as my-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-cc
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   where

   <initdata_string>

   Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.

   <container_name>

   Specify a container name.

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: "<initdata_string>"
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:latest
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault
   metadata:
     name: coco-test-pod
     labels:
       app: coco-test-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: "<initdata_string>"
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: test-container
         image: registry.access.redhat.com/ubi9/ubi:9.3
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   where:

   io.katacontainers.config.hypervisor.cc_init_data

   Optional: Specifies initdata in a pod annotation, which overrides the global INITDATA setting in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

3. Log in to the pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

4. Fetch the Red Hat build of Trustee resource by running the following command:

   $ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status

   Example output

   success #/

Procedure

1. Add spec.runtimeClassName: kata-cc to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-cc
   # ...

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.12.0

2. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

3. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

4. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION   REPLACES    PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.12.0    1.11.1      Succeeded

Procedure

1. Search for the pods by running the following command:

   $ oc get pods -A -o json | jq -r '.items[] | \
     select(.spec.runtimeClassName == "kata-cc").metadata.name'

2. Delete each pod by running the following command:

   $ oc delete pod <pod>

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

   The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

   Important

   When you delete the KataConfig CR, the CLI stops responding until all worker nodes reboot. You must wait for the deletion process to complete before performing the verification.

2. Verify the CR removal by running the following command:

   $ oc get kataconfig example-kataconfig

   Example output

   No example-kataconfig instances exist

Procedure

1. Delete the subscription by running the following command:

   $ oc delete subscription OpenShift sandboxed containers Operator -n openshift-sandboxed-containers-operator

2. Delete the namespace by running the following command:

   $ oc delete namespace openshift-sandboxed-containers-operator

Procedure

1. Delete the KataConfig CRD by running the following command:

   $ oc delete crd kataconfigs.kataconfiguration.openshift.io

2. Verify that the CRD was deleted by running the following command:

   $ oc get crd kataconfigs.kataconfiguration.openshift.io

   Example output

   Unknown CRD kataconfigs.kataconfiguration.openshift.io

Procedure

1. In the OpenShift Container Platform web console, navigate to Observe → Metrics.

2. In the input field, enter the query for the metric you want to observe.

   All kata-related metrics begin with kata. Typing kata displays a list of all available kata metrics.

Procedure

1. Change the logLevel field in your existing KataConfig CR to debug:

   $ oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}'

2. Monitor the kata-oc machine config pool until the value of UPDATED is True, indicating that all worker nodes are updated:

   $ oc get mcp kata-oc

   Example output

   NAME     CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
   kata-oc  rendered-kata-oc-169   False    True      False     3             1                  1                    0                     9h

Verification

1. Start a debug session with a node in the machine config pool:

   $ oc debug node/<node_name>

2. Change the root directory to /host:

   # chroot /host

3. Verify the changes in the crio.conf file:

   # crio config | egrep 'log_level

   Example output

   log_level = "debug"