Deploying confidential containers

1. Ensure that you are logged in to Jira. If you do not have a Jira account, you must create a Red Hat Jira account.
2. Launch the Create Issue form.

3. Complete the Summary, Description, and Reporter fields.

   In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue.

4. Click Create.

1. Create MachineConfig for Intel® Trust Domain Extensions (TDX).
2. Install the OpenShift sandboxed containers Operator.
3. Configure the remote attestation infrastructure for Intel® TDX workloads.
4. Enable the confidential containers feature gate.
5. Create the KataConfig CR.
6. Verify the attestation process.

Procedure

1. Create a tdx-machine-config.yaml manifest file according to the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: <role> 

   1

   
     name: 99-enable-intel-tdx
   spec:
     kernelArguments:
     - kvm_intel.tdx=1
     - nohibernate
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modules-load.d/vsock.conf
             mode: 0644
             contents:
               source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

   Copy to Clipboard Toggle word wrap

   1

   Specify master for single-node OpenShift or kata-oc for a multi-node cluster.

2. Create the config map by running the following command:

   $ oc create -f tdx-machine-config.yaml

   Copy to Clipboard Toggle word wrap

   Updating the machine config triggers node reboot.

Verification

1. Verify that the machine config is correctly configured by running the following command:

   $ oc get oc get machineconfig 99-enable-intel-tdx

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.10.3        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure the remote attestation project:

   1. Create the intel-dcap namespace by running the following command:

      $ oc create namespace intel-dcap

      Copy to Clipboard Toggle word wrap

   2. Switch to the intel-dcap project by running the following command:

      $ oc project intel-dcap

      Copy to Clipboard Toggle word wrap

   3. Update the Security Context Constraint by running the following command:

      $ oc adm policy add-scc-to-user privileged -z default

      Copy to Clipboard Toggle word wrap

2. Switch to the default project by running the following command:

   $ oc project default

   Copy to Clipboard Toggle word wrap

3. Set the PCCS variables by running the following commands:

   $ export PCCS_API_KEY="${PCCS_API_KEY:-}"

   Copy to Clipboard Toggle word wrap

   To obtain the API key for the Intel® Software Guard Extensions and Intel® TDX Provisioning Certification Service, navigate to the Intel Trusted Services API portal, sign in, and subscribe to the Provisioning Certification Service. The API key is displayed on the Manage Subscriptions page.

   $ export PCCS_USER_TOKEN="${PCCS_USER_TOKEN:-mytoken}"

   Copy to Clipboard Toggle word wrap

   For details about PCCS tokens, see the Design Guide for Intel® SGX Provisioning Certificate Caching Service (Intel® SGX PCCS).

   $ export PCCS_ADMIN_TOKEN="${PCCS_ADMIN_TOKEN:-mytoken}"

   Copy to Clipboard Toggle word wrap

   $ export PCCS_NODE=$(oc get nodes \
     -l 'node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=' \
     -o jsonpath='{.items[0].metadata.name}')

   Copy to Clipboard Toggle word wrap

4. Set the cluster proxy variable by running the appropriate command:

   $ export CLUSTER_HTTPS_PROXY="$(oc get proxy/cluster \
     -o jsonpath={.spec.httpsProxy})"

   Copy to Clipboard Toggle word wrap

   $ export CLUSTER_NO_PROXY="$(oc get proxy/cluster \
     -o jsonpath={.spec.noProxy})"

   Copy to Clipboard Toggle word wrap

5. Create the PCCS secrets:

   1. Set the PCCS secrets variables by running the following commands:

      $ export PCCS_USER_TOKEN_HASH=$(echo -n "$PCCS_USER_TOKEN" | sha512sum | tr -d '[:space:]-')

      Copy to Clipboard Toggle word wrap

      $ export PCCS_ADMIN_TOKEN_HASH=$(echo -n "$PCCS_ADMIN_TOKEN" | sha512sum | tr -d '[:space:]-')

      Copy to Clipboard Toggle word wrap

      $ PCCS_PEM_CERT_PATH=$(mktemp -d)

      Copy to Clipboard Toggle word wrap
      Note

      This directory is automatically deleted at reboot. To re-use the PCCS certificate and key, you must create a persistent directory.

   2. Generate an RSA key pair and output the private key as a PCCS certificate by running the following command:

      $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
        -keyout $PCCS_PEM_CERT_PATH/private.pem \
        -out $PCCS_PEM_CERT_PATH/certificate.pem \
        -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

      Copy to Clipboard Toggle word wrap

   3. Set the PCCS certificate variables by running the following commands:

      $ export PCCS_PEM=$(cat "$PCCS_PEM_CERT_PATH"/private.pem | base64 | tr -d '\n')

      Copy to Clipboard Toggle word wrap

      $ export PCCS_CERT=$(cat "$PCCS_PEM_CERT_PATH"/certificate.pem | base64 | tr -d '\n')

      Copy to Clipboard Toggle word wrap

   4. Create the PCCS secrets by running the following command:

      $ oc create secret generic pccs-secrets \
          --namespace intel-dcap \
          --from-literal=PCCS_API_KEY="$PCCS_API_KEY" \
          --from-literal=PCCS_USER_TOKEN_HASH="$PCCS_USER_TOKEN_HASH" \
          --from-literal=USER_TOKEN="$PCCS_USER_TOKEN" \
          --from-literal=PCCS_ADMIN_TOKEN_HASH="$PCCS_ADMIN_TOKEN_HASH"

      Copy to Clipboard Toggle word wrap

6. Create the PCCS by running the following command:

   $ oc apply -f <(curl -sSf https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/refs/heads/devel/scripts/install-helpers/baremetal-coco/intel-dcap/pccs.yaml.in|envsubst)

   Copy to Clipboard Toggle word wrap

7. Create the QGS by running the following command:

   $ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/refs/heads/devel/scripts/install-helpers/baremetal-coco/intel-dcap/qgs.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"
     deploymentMode: <deployment_mode>

   Copy to Clipboard Toggle word wrap

   where

   <deployment_mode>

   On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:

   • MachineConfig for clusters that always use the MCO
   • DaemonSet for clusters that never use the MCO
   • DaemonSetFallback for clusters that sometimes use the MCO

2. Create the my-feature-gates config map by running the following command:

   $ oc create -f my-feature-gate.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the initdata.toml file:

   algorithm = "sha384"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   '''
   
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   [image]
   image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
   '''
   
   "policy.rego" = '''
   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }
   '''

   Copy to Clipboard Toggle word wrap

   url

   Specify the Red Hat build of Trustee

   <kbs_certificate>

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

   image_security_policy_uri

   Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

2. Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Copy to Clipboard Toggle word wrap

   Record this string to use in the pod manifest.

3. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

   $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)

   Copy to Clipboard Toggle word wrap

4. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

   Copy to Clipboard Toggle word wrap

5. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Copy to Clipboard Toggle word wrap

   Record the PCR8_HASH value for the RVPS config map.

Procedure

1. Add the initdata string to the pod manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-cc
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
   
     enablePeerPods: false
     checkNodeEligibility: true
   
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.

   Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.

4. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-cc      kata-tdx      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 

   1

   
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:latest
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault
   metadata:
     name: coco-test-pod
     labels:
       app: coco-test-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 

   2

   
   spec:
     runtimeClassName: kata-cc
     containers:
       - name: test-container
         image: registry.access.redhat.com/ubi9/ubi:9.3
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

   1 1 2

   Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

   Copy to Clipboard Toggle word wrap

3. Log in to the pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

   Copy to Clipboard Toggle word wrap

4. Fetch the Red Hat build of Trustee resource by running the following command:

   $ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status

   Copy to Clipboard Toggle word wrap

   Example output

   success #/

   Copy to Clipboard Toggle word wrap

1. Prepare your network by configuring outbound connectivity for the peer pods.
2. Install the OpenShift sandboxed containers Operator.
3. Enable the confidential containers feature gate.
4. Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.
5. Create the peer pods config map. You can add initdata to the config map to create a default global configuration for your peer pods.
6. Create the KataConfig CR.
7. Verify the attestation process.
8. You can add initdata to a pod manifest to override the global initdata configuration you set in the peer pods config map.
9. Optional: If you select a container image from an authenticated registry, you must configure a pull secret for the pod.
10. Optional: You can select a custom peer pod VM image.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.10.3        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"
     deploymentMode: <deployment_mode>

   Copy to Clipboard Toggle word wrap

   where

   <deployment_mode>

   On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:

   • MachineConfig for clusters that always use the MCO
   • DaemonSet for clusters that never use the MCO
   • DaemonSetFallback for clusters that sometimes use the MCO

2. Create the my-feature-gates config map by running the following command:

   $ oc create -f my-feature-gate.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Obtain the Red Hat build of Trustee URL by running the following command:

   $ TRUSTEE_URL=$(oc get route kbs-service \
     -n trustee-operator-system -o jsonpath='{.spec.host}') \
     && echo $TRUSTEE_URL

   Copy to Clipboard Toggle word wrap

2. Create the initdata.toml file:

   algorithm = "sha384"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   '''
   
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   [image]
   image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
   '''
   
   "policy.rego" = '''
   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }
   '''

   Copy to Clipboard Toggle word wrap

   url

   Specify the Red Hat build of Trustee URL. If you configure the Red Hat build of Trustee with insecure_http for testing purposes, use HTTP. Otherwise, use HTTPS. For production systems, avoid using insecure_http unless you configure your environment to handle TLS externally, for example, with a proxy.

   <kbs_certificate>

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

   image_security_policy_uri

   Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

3. Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Copy to Clipboard Toggle word wrap

   Record this string to use in the peer pods config map or the peer pod manifest.

4. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

   $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)

   Copy to Clipboard Toggle word wrap

5. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

   Copy to Clipboard Toggle word wrap

6. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Copy to Clipboard Toggle word wrap

   Record the PCR8_HASH value for the RVPS config map.

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
        && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

      Copy to Clipboard Toggle word wrap

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Name:name}" --output tsv)

      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
        --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
         && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

      Copy to Clipboard Toggle word wrap

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

      Copy to Clipboard Toggle word wrap

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "{Location:location}" --output tsv) \
        && echo "AZURE_REGION: \"$AZURE_REGION\""

      Copy to Clipboard Toggle word wrap

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     AZURE_INSTANCE_SIZE: "Standard_DC2as_v5"
     AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5"
     AZURE_SUBNET_ID: "<azure_subnet_id>"
     AZURE_NSG_ID: "<azure_nsg_id>"
     AZURE_IMAGE_ID: ""
     AZURE_REGION: "<azure_region>"
     AZURE_RESOURCE_GROUP: "<azure_resource_group>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "false"
     INITDATA: "<initdata_string>"

   Copy to Clipboard Toggle word wrap

   AZURE_INSTANCE_SIZE

   Defines the default instance size that is used if the instance size is not defined in the workload object. "Standard_DC2as_v5" is for AMD SEV-SNP. If your TEE is Intel® Trust Domain Extensions (TDX), specify Standard_EC4eds_v5.

   AZURE_IMAGE_ID

   Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod VM image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster. This image provides root disk integrity protection (dm-verity) and encrypted container storage. See Confidential VMs: The core of confidential containers for details.

   AZURE_INSTANCE_SIZES

   Specify the allowed instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

   INITDATA

   Specify the initdata string to create a default configuration for all peer pods. If you add initdata to a peer pod manifest, that setting overrides this global configuration.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add the initdata string to the pod manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
   
     enablePeerPods: true
   
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard Toggle word wrap

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 

   1

   
   spec:
     runtimeClassName: kata-remote
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:latest
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

   1

   Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

   Copy to Clipboard Toggle word wrap

3. Log in to the pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

   Copy to Clipboard Toggle word wrap

4. Fetch the Red Hat build of Trustee resource by running the following command:

   $ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status

   Copy to Clipboard Toggle word wrap

   Example output

   success #/

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy to Clipboard Toggle word wrap

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy to Clipboard Toggle word wrap

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy to Clipboard Toggle word wrap

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap

2. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy to Clipboard Toggle word wrap

1. Install the OpenShift sandboxed containers Operator.
2. Create the peer pods secret.
3. Enable the confidential containers feature gate.
4. Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.

5. Create initdata to initialize a pod with sensitive or workload-specific data at runtime.

   Important

   Do not use the default permissive Kata Agent policy in a production environment. You must configure a restrictive policy, preferably by creating initdata.

   As a minimum requirement, you must disable ExecProcessRequest to prevent a cluster administrator from accessing sensitive data by running the oc exec command on a confidential containers pod.

6. Create the peer pods config map. You can add initdata to the config map to create a default global configuration for your peer pods.
7. Create the KataConfig CR.
8. Verify the attestation process.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Manual
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Get the InstallPlan CR for the OpenShift sandboxed containers Operator by running the following command:

   $ oc get installplan -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   • Installation example output

     NAME            CSV                                      APPROVAL  APPROVED
     install-bl4fl   sandboxed-containers-operator.v1.11.1    Manual    false

     Copy to Clipboard Toggle word wrap

   • Upgrade example output

     NAME            CSV                                     APPROVAL   APPROVED
     install-jdzrb   sandboxed-containers-operator.v1.11.1   Manual     false
     install-pfk8l   sandboxed-containers-operator.v1.10.3   Manual     true

     Copy to Clipboard Toggle word wrap

8. Approve the manual installation by running the following command:

   $ oc patch installplan <installplan_name> -p '{"spec":{"approved":true}}' --type=merge -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   <installplan_name>

   Specify the InstallPlan resource. For example, install-jdzrb.

9. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

10. Watch the process by running the following command:

    $ watch oc get csv -n openshift-sandboxed-containers-operator

    Copy to Clipboard Toggle word wrap

    Example output

    NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.10.3        Succeeded

    Copy to Clipboard Toggle word wrap