Chapter 3. Deploying confidential containers on IBM Z and IBM LinuxONE

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_URI: "<libvirt_gateway_uri>" 

   1

   
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 

   2

   
     HOST_KEY_CERTS: "<host_key_crt_value>" 

   3

   1

   Specify the libvirt URI.

   2

   Specify the Red Hat offline token, which is required for the Operator-built image.

   3

   Specify the HKD certificate value to enable IBM Secure Execution for the Operator-built image.

2. Create the secret by running the following command:

   $ oc create -f peer-pods-secret.yaml

Procedure

1. Create a cc-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"

2. Create the osc-feature-gates config map by running the following command:

   $ oc create -f cc-feature-gate.yaml

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

Procedure

1. Obtain the Red Hat build of Trustee IP address by running the following command:

   $ oc get node $(oc get pod -n trustee-operator-system \
     -o jsonpath='{.items[0].spec.nodeName}') \
     -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

   Example output

   192.168.122.22

2. Obtain the port by running the following command:

   $ oc get svc kbs-service -n trustee-operator-system

   Example output

   NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
   kbs-service  NodePort    172.30.116.11   <none>        8080:32178/TCP   12d

3. Create the initdata.toml file:

   algorithm = "sha384"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   
   url = '<trustee_url>'
   
   [token_configs.kbs]
   url = '<trustee_url>'
   cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   '''
   
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '<trustee_url>'
   kbs_cert = """
   -----BEGIN CERTIFICATE-----
   <kbs_certificate>
   -----END CERTIFICATE-----
   """
   '''
   
   "policy.rego" = '''
   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := true
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true
   '''

   URL

   Specify the Red Hat build of Trustee IP address and the port, for example, https://192.168.122.22:32178.

   <kbs_certificate>

   Specify the Base64-encoded TLS certificate for the attestation agent.

   kbs_cert

   Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

4. Convert the initdata.toml file to a Base64-encoded string in gzip format in a text file by running the following command:

   $ cat initdata.toml | gzip | base64 -w0 > initdata.txt

   Record this string for the peer pods config map or a peer pod manifest.

5. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

   $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)

6. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

7. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Record the PCR8_HASH value for the. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

   $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)

8. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

   $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

9. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

   $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH

   Record the PCR8_HASH value for the RVPS config map.

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_POOL: "<libvirt_pool>"
     LIBVIRT_VOL_NAME: "<libvirt_volume>"
     LIBVIRT_DIR_NAME: "/var/lib/libvirt/images/<directory_name>"
     LIBVIRT_NET: "default"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "false"
     INITDATA: "<initdata_string>"

   LIBVIRT_POOL

   If you have manually configured the libvirt pool, use the same name as in your KVM host configuration.

   LIBVIRT_VOL_NAME

   If you have manually configured the libvirt volume, use the same name as in your KVM host configuration.

   LIBVIRT_DIR_NAME

   Specify the libvirt directory for storing virtual machine disk images, such as .qcow2, or .raw files. To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.

   LIBVIRT_NET

   Specify a libvirt network if you do not want to use the default network.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

   INITDATA

   Specify the initdata string to create a default configuration for all peer pods. If you add initdata to a peer pod manifest, that setting overrides this global configuration.

2. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

Procedure

1. Add the initdata string to the pod manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: ocp-cc-pod
     labels:
       app: ocp-cc-pod
     annotations:
       io.katacontainers.config.runtime.cc_init_data: <initdata_string>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

2. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

Procedure

1. Set the LIBVIRT_POOL variable by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

2. Set the LIBVIRT_VOL_NAME variable to a new libvirt volume by running the following command:

   $ export LIBVIRT_VOL_NAME=<new_libvirt_volume>

3. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

4. Upload the custom peer pod VM image to the new libvirt volume:

   $ virsh -c qemu:///system vol-upload \
     --vol $LIBVIRT_VOL_NAME <custom_podvm_image.qcow2> \
     --pool $LIBVIRT_POOL --sparse

5. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<new_libvirt_volume>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

6. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   DISABLECVM: false 

   1

   
   LIBVIRT_IMAGE_ID: fa-pp-vol 

   2

   
   
   BinaryData
   ====
   Events: <none>

   1

   Enables the Confidential VM during IBM Secure Execution for the Operator-built image.

   2

   Contains a value if you have built the peer pod image and uploaded it to the libvirt volume.

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

Procedure

1. Create a test-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: busybox
     namespace: default
     annotations:
       io.katacontainers.config.runtime.cc_init_data: <initdata_string>
     labels:
       run: busybox
   spec:
     runtimeClassName: kata-remote
     restartPolicy: Never
     containers:
     - name: busybox
       image: quay.io/prometheus/busybox:latest
       imagePullPolicy: Always
       command:
         - "sleep"
         - "3600"

2. Create the pod by running the following command:

   $ oc create -f test-pod.yaml

3. Log in to the pod by running the following command:

   $ oc exec -it busybox -n default -- /bin/sh

4. Fetch the pod secret by running the following command:

   $ wget http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

   Example output

   Connecting to 127.0.0.1:8006 (127.0.0.1:8006)
   saving to 'key1'
   key1                 100% |*******************************************|     8  0:00:00 ETA
   'key1' saved

5. Display the key1 value by running the following command:

   $ cat key1

   Example output

   res1val1/ #