Chapter 2. Deploying Confidential Containers on Azure

Procedure

1. Create a cc-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"

   Copy to Clipboard

2. Create the config map by running the following command:

   $ oc apply -f cc-feature-gate.yaml

   Copy to Clipboard

Procedure

1. Obtain the Trustee IP address and port. In production environments, Trustee is deployed on a separate OpenShift Container Platform cluster running on trusted hardware, distinct from the cloud environment hosting the primary workload. Run the following commands on the Trustee cluster.

2. Create the initdata.toml configuration file:

   ```toml
   algorithm = "sha384"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   url = '<url>:<port>' 

   1

   
   
   
   [token_configs.kbs]
   url = '<url>:<port>'
   cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate> 

   2

   
   -----END CERTIFICATE-----
   """
   '''
   
   "cdh.toml"  = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<url>:<port>'
   kbs_cert = """ 

   3

   
   -----BEGIN CERTIFICATE-----
   <kbs_certificate> 

   4

   
   -----END CERTIFICATE-----
   """
   '''
   
   "policy.rego" = ''' 

   5

   
   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := false
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true
   '''
   ```

   Copy to Clipboard

   1

   Specify the URL and port of the Trustee instance. If you configure the Trustee with insecure_http for testing purposes, use HTTP. Otherwise, use HTTPS. For production systems, avoid using insecure_http unless you configure your environment to handle TLS externally, for example, with a proxy.

   2

   Specify the Base64-encoded TLS certificate for the attestation agent. This is not required for testing purposes, but it is recommended for production systems.

   3

   Delete the kbs_cert setting if you configure insecure_http = true in the Trustee config map.

   4

   Specify the Base64-encoded TLS certificate for the Trustee instance.

   5

   Specify a custom Kata Agent policy. The default policy allows all API calls. For production environments, set ReadStreamRequest and ExecProcessRequest to false to disable the oc exec and oc log APIs, preventing unencrypted data transmission via the control plane. Adjust other true or false values to customize the policy further based on your needs.

3. Convert the initdata.toml file to a Base64-encoded string in gzip format in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Copy to Clipboard

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

      Copy to Clipboard

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      Copy to Clipboard

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

      Copy to Clipboard

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

      Copy to Clipboard

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""

      Copy to Clipboard

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     AZURE_INSTANCE_SIZE: "Standard_DC2as_v5" 

   1

   
     AZURE_INSTANCE_SIZES: "Standard_DC2as_v5, Standard_DC4as_v5, Standard_DC8as_v5" 

   2

   
     AZURE_SUBNET_ID: "<azure_subnet_id>" 

   3

   
     AZURE_NSG_ID: "<azure_nsg_id>" 

   4

   
     PROXY_TIMEOUT: "5m"
     AZURE_IMAGE_ID: "<azure_image_id>" 

   5

   
     AZURE_REGION: "<azure_region>" 

   6

   
     AZURE_RESOURCE_GROUP: "<azure_resource_group>" 

   7

   
     PEERPODS_LIMIT_PER_NODE: "10" 

   8

   
     TAGS: "key1=value1,key2=value2" 

   9

   
     ROOT_VOLUME_SIZE: "6" 

   10

   
     INITDATA: "<base64_encoded_initdata>" 

   11

   
     DISABLECVM: "false"

   Copy to Clipboard

   1

   The "Standard_DC2as_v5" value is the default if an instance size is not defined in the workload. Ensure the instance type supports the trusted environment. The default "Standard_DC2as_v5" value is for AMD SEV-SNP. If your TEE is Intel TDX, specify Standard_EC4eds_v5.

   2

   Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads. For Intel TDX, specify "Standard_EC4eds_v5, Standard_EC8eds_v5, Standard_EC16eds_v5".

   3

   Specify the AZURE_SUBNET_ID value that you retrieved.

   4

   Specify the AZURE_NSG_ID value that you retrieved.

   5

   Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.

   6

   Specify the AZURE_REGION value you retrieved.

   7

   Specify the AZURE_RESOURCE_GROUP value you retrieved.

   8

   Specify the maximum number of peer pods that can be created per node. The default value is 10.

   9

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   10

   Optional: Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB. Increase this value for pods with larger container images.

   11

   Specify the Base64-encoded string in gzip format created in the initdata.txt file.

3. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

   Copy to Clipboard

4. Restart the ds/osc-caa-ds daemon set by running the following command:

   $ oc set env ds/osc-caa-ds \
     -n openshift-sandboxed-containers-operator REBOOT="$(date)"

   Copy to Clipboard

Procedure

1. Create an initdata string for the pod.

2. Add the Base64-encoded initdata a my-pod.yaml pod manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.runtime.cc_init_data: <base64_initdata>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard

3. Apply the pod manifest by running the following command:

   $ oc apply -f my-pod.yaml

   Copy to Clipboard

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

   Copy to Clipboard

   The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

   Important

   When you delete the KataConfig CR, the CLI stops responding until all worker nodes reboot. You must wait for the deletion process to complete before performing the verification.

2. Verify the CR removal by running the following command:

   $ oc get kataconfig example-kataconfig

   Copy to Clipboard

   Example output

   No example-kataconfig instances exist

   Copy to Clipboard

Procedure

1. Edit the pod manifest by adding the io.katacontainers.config.hypervisor.image annotation and save it in a pod-manifest.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>" 

   1

   
   spec:
     runtimeClassName: kata-remote 

   2

   
     containers:
     - name: <example_container> 

   3

   
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard

   1

   Specify the custom peer pod image ID.

   2

   Ensure that the runtimeClassName field is set to kata-remote to create a peer pod.

   3

   Specify the container name.

2. Create the pod by running the following command:

   $ oc apply -f pod-manifest.yaml

   Copy to Clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard

   1

   Optional: If you have applied node labels to install {runtime} on specific nodes, specify the key and value, for example, cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   Copy to Clipboard

   The new KataConfig CR is created and installs {runtime} as a runtime class on the worker nodes.

   Wait for the {runtime} installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the {runtime} is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

   Copy to Clipboard

Procedure

1. Create a verification-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.runtime.cc_init_data: <base64_initdata> 

   1

   
   spec:
     runtimeClassName: kata-remote
     containers:
       - name: skr-openshift
         image: registry.access.redhat.com/ubi9/ubi:9.3
         command:
           - sleep
           - "36000"
         securityContext:
           privileged: false
           seccompProfile:
             type: RuntimeDefault

   Copy to Clipboard

   1

   This pod annotation overrides the Base64-encoded string of the initdata.toml file set globally in the peer pods config map.

2. Create the pod by running the following command:

   $ oc create -f verification-pod.yaml

   Copy to Clipboard

3. Connect to the Bash shell of the ocp-cc-pod by running the following command:

   $ oc exec -it ocp-cc-pod -- bash

   Copy to Clipboard

4. Fetch the pod secret by running the following command:

   $ curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

   Copy to Clipboard

   Example output

   res1val1

   Copy to Clipboard

   The Trustee server returns the secret only if the attestation is successful.