Chapter 5. Deploying on IBM Z and IBM LinuxONE

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.7.0

6. Create the subscription by running the following command:

   $ oc apply -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.7.0    1.6.0        Succeeded

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   -o jsonpath='{.spec.limit}{"\n"}'

2. Modify the limit attribute of the peerpodConfig CR by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
   --type merge --patch '{"spec":{"limit":"<value>"}}' 

   1

   1

   Replace <value> with the limit you want to define.

Procedure

1. Log in to the KVM host.

2. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   You need the LIBVIRT_POOL value to create the secret for the libvirt provider.

3. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_VOL_NAME=<libvirt_volume>

   You need the LIBVIRT_VOL_NAME value to create the secret for the libvirt provider.

4. Set the path of the default storage pool location, by running the following command:

   $ export LIBVIRT_POOL_DIRECTORY=<target_directory> 

   1

   1

   To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.

5. Create a libvirt pool by running the following command:

   $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

6. Start the libvirt pool by running the following command:

   $ virsh pool-start $LIBVIRT_POOL

7. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

Procedure

1. Create a Dockerfile.podvm-oci file:

   FROM scratch
   
   ARG PODVM_IMAGE_SRC
   ENV PODVM_IMAGE_PATH="/image/podvm.qcow2"
   
   COPY $PODVM_IMAGE_SRC $PODVM_IMAGE_PATH

2. Build a container with the pod VM QCOW2 image by running the following command:

   $ docker build -t podvm-libvirt \
     --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 

   1

   
     --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 

   2

   
     -f Dockerfile.podvm-oci .

   1

   Specify the QCOW2 image source on the host.

   2

   Optional: Specify the path of the QCOW2 image if you do not use the default, /image/podvm.qcow2.

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_URI: "<libvirt_gateway_uri>" 

   1

   
     LIBVIRT_POOL: "<libvirt_pool>" 

   2

   
     LIBVIRT_VOL_NAME: "<libvirt_volume>" 

   3

   
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 

   4

   1

   Specify the libvirt URI.

   2

   Specify the libvirt pool.

   3

   Specify the libvirt volume name.

   4

   Specify the Red Hat offline token, which is required for the Operator-built image.

2. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     DISABLECVM: "true"

2. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

Procedure

1. Create a libvirt-podvm-image-cm.yaml manifest according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: libvirt-podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     PODVM_DISTRO: "rhel"
     CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor"
     CAA_REF: "<cloud_api_adaptor_version>" 

   1

   
     DOWNLOAD_SOURCES: "no"
     CONFIDENTIAL_COMPUTE_ENABLED: "yes"
     UPDATE_PEERPODS_CM: "yes"
     ORG_ID: "<rhel_organization_id>"
     ACTIVATION_KEY: "<rhel_activation_key>" 

   2

   
     IMAGE_NAME: "<podvm_libvirt_image>"
     PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 

   3

   
     SE_BOOT: "true" 

   4

   
     BASE_OS_VERSION: "<rhel_image_os_version>" 

   5

   1

   Specify the latest version of the Cloud API Adaptor source.

   2

   Specify your RHEL activation key.

   3

   Optional: Specify the following values if you created a container image:

   • image_repo_url: Container registry URL.
   • image_tag: Image tag.
   • image_path: Image path. Default: /image/podvm.qcow2.

   4

   SE_BOOT: "true" enables IBM Secure Execution for an Operator-built image. Set to false if you created a container image.

   5

   Specify the RHEL image operating system version. IBM Z® Secure Execution supports RHEL 9.4 and later versions.

2. Create the config map by running the following command:

   $ oc apply -f libvirt-podvm-image-cm.yaml

   The libvirt pod VM image config map is created for your libvirt provider.

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

2. Copy the public SSH key to your KVM host:

   $ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP>

3. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

4. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   
   BinaryData
   ====
   Events: <none>

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. Create a trustee-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: trustee-operator-system

2. Create the trustee-operator-system namespace by running the following command:

   $ oc apply -f trustee-namespace.yaml

3. Create a trustee-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: trustee-operator-group
     namespace: trustee-operator-system
   spec:
     targetNamespaces:
     - trustee-operator-system

4. Create the operator group by running the following command:

   $ oc apply -f trustee-operatorgroup.yaml

5. Create a trustee-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: trustee-operator
     namespace: trustee-operator-system
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: trustee-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: trustee-operator.v0.1.0

6. Create the subscription by running the following command:

   $ oc apply -f trustee-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n trustee-operator-system

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n trustee-operator-system

   Example output

   NAME                      DISPLAY                        PHASE
   trustee-operator.v0.1.0   Trustee Operator  0.1.0        Succeeded

Procedure

1. Create an edge route by running the following command:

   $ oc create route edge --service=kbs-service --port kbs-port \
     -n trustee-operator-system

   Note

   Note: Currently, only a route with a valid CA-signed certificate is supported. You cannot use a route with self-signed certificate.

2. Set the TRUSTEE_HOST variable by running the following command:

   $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
     -o jsonpath={.spec.host})

3. Verify the route by running the following command:

   $ echo $TRUSTEE_HOST

   Example output

   kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io

Procedure

1. Create a cc-feature-gate.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: osc-feature-gates
     namespace: openshift-sandboxed-containers-operator
   data:
     confidential: "true"

2. Create the config map by running the following command:

   $ oc apply -f cc-feature-gate.yaml

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     DISABLECVM: "false"

2. Create the config map by running the following command:

   $ oc apply -f peer-pods-cm.yaml

3. Restart the peerpodconfig-ctrl-caa-daemon daemon set by running the following command:

   $ oc set env ds/peerpodconfig-ctrl-caa-daemon \
     -n openshift-sandboxed-containers-operator REBOOT="$(date)"

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

2. Verify that the custom resource was deleted by running the following command:

   $ oc get kataconfig example-kataconfig

   Example output

   No example-kataconfig instances exist

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 

   1

   
     HKD_CRT: "<hkd_crt_value>" 

   2

   1

   Specify the Red Hat offline token, which is required for the Operator-built image.

   2

   Specify the HKD certificate value to enable IBM Secure Execution for the Operator-built image.

2. Create the secret by running the following command:

   $ oc apply -f peer-pods-secret.yaml

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc apply -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   DISABLECVM: false 

   1

   
   LIBVIRT_IMAGE_ID: fa-pp-vol 

   2

   
   
   BinaryData
   ====
   Events: <none>

   1

   Enables the Confidential VM during IBM Secure Execution for the Operator-built image.

   2

   Contains a value if you have built the peer pod image and uploaded it to the libvirt volume.

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata             kata             152m
   kata-remote      kata-remote      152m

Procedure

1. Create a private key by running the following command:

   $ openssl genpkey -algorithm ed25519 > privateKey

2. Create a public key by running the following command:

   $ openssl pkey -in privateKey -pubout -out publicKey

3. Create a secret by running the following command:

   $ oc create secret generic kbs-auth-public-key --from-file=publicKey -n trustee-operator-system

4. Verify the secret by running the following command:

   $ oc get secret -n trustee-operator-system

Procedure

1. Create a kbs-config-cm.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kbs-config-cm
     namespace: trustee-operator-system
   data:
     kbs-config.json: |
       {
         "insecure_http" : true,
         "sockets": ["0.0.0.0:8080"],
         "auth_public_key": "/etc/auth-secret/publicKey",
         "attestation_token_config": {
           "attestation_token_type": "CoCo"
         },
         "repository_config": {
           "type": "LocalFs",
           "dir_path": "/opt/confidential-containers/kbs/repository"
         },
         "as_config": {
           "work_dir": "/opt/confidential-containers/attestation-service",
           "policy_engine": "opa",
           "attestation_token_broker": "Simple",
             "attestation_token_config": {
             "duration_min": 5
             },
           "rvps_config": {
             "store_type": "LocalJson",
             "store_config": {
               "file_path": "/opt/confidential-containers/rvps/reference-values/reference-values.json"
             }
            }
         },
         "policy_engine_config": {
           "policy_path": "/opt/confidential-containers/opa/policy.rego"
         }
       }

2. Create the config map by running the following command:

   $ oc apply -f kbs-config-cm.yaml

Procedure

1. Create a temporary folder for the SE header by running the following command:

   $ mkdir -p /tmp/ibmse/hdr

2. Download the pvextract-hdr tool from IBM s390 Linux repository by running the following command:

   $ wget https://github.com/ibm-s390-linux/s390-tools/raw/v2.33.1/rust/pvattest/tools/pvextract-hdr -O /tmp/pvextract-hdr

3. Make the tool executable by running the following command:

   $ chmod +x /tmp/pvextract-hdr

4. Set the $IMAGE_OUTPUT_DIR variable by running the following command:

   $ export IMAGE=$IMAGE_OUTPUT_DIR/se-podvm-commit-short-id.qcow2

5. Set the $IMAGE variable by running the following command:

   $ export IMAGE=/root/rooo/se-podvm-d1fb986-dirty-s390x.qcow2

6. Enable the nbd kernel module by running the following command:

   $ modprobe nbd

7. Connect the SE image as a network block device (NBD) by running the following command:

   $ qemu-nbd --connect=/dev/nbd0 $IMAGE

8. Create a mount directory for the SE image by running the following command:

   $ mkdir -p /mnt/se-image/

9. Pause the process by running the following command:

   $ sleep 1

10. List your block devices by running the following command:

    $ lsblk

    Example output

    nbd0                                           43:0    0  100G  0 disk
    ├─nbd0p1                                       43:1    0  255M  0 part
    ├─nbd0p2                                       43:2    0    6G  0 part
    │ └─luks-e23e15fa-9c2a-45a5-9275-aae9d8e709c3 253:2    0    6G  0 crypt
    └─nbd0p3                                       43:3    0 12.4G  0 part
    nbd1                                           43:32   0   20G  0 disk
    ├─nbd1p1                                       43:33   0  255M  0 part
    ├─nbd1p2                                       43:34   0    6G  0 part
    │ └─luks-5a540f7c-c0cb-419b-95e0-487670d91525 253:3    0    6G  0 crypt
    └─nbd1p3                                       43:35   0 86.9G  0 part
    nbd2                                           43:64   0    0B  0 disk
    nbd3                                           43:96   0    0B  0 disk
    nbd4                                           43:128  0    0B  0 disk
    nbd5                                           43:160  0    0B  0 disk
    nbd6                                           43:192  0    0B  0 disk
    nbd7                                           43:224  0    0B  0 disk
    nbd8                                           43:256  0    0B  0 disk
    nbd9                                           43:288  0    0B  0 disk
    nbd10                                          43:320  0    0B  0 disk

11. Mount the SE image directory on an available NBD partition and extract the SE header by running the following command:

    $ mount /dev/<nbdXp1> /mnt/se-image/ /tmp/pvextract-hdr \
      -o /tmp/ibmse/hdr/hdr.bin /mnt/se-image/se.img

    Example output

    SE header found at offset 0x014000
    SE header written to '/tmp/ibmse/hdr/hdr.bin' (640 bytes)

    The following error is displayed if the NBD is unavailable:

    mount: /mnt/se-image: can't read superblock on /dev/nbd0p1

12. Unmount the SE image directory by running the following command:

    $ umount /mnt/se-image/

13. Disconnect the network block storage device by running the following command:

    $ qemu-nbd --disconnect /dev/nbd0

Procedure

1. Obtain the attestation policy fields by performing the following steps:

   1. Download the se_parse_hdr.py script from the OpenShift Trustee repository by running the following command:

      $ wget https://github.com/openshift/trustee/raw/main/attestation-service/verifier/src/se/se_parse_hdr.py -O /tmp/se_parse_hdr.py

   2. Create a temporary directory for the SE Host Key Document (HKD) certificate by running the following command:

      $ mkdir /tmp/ibmse/hkds/

   3. Copy your Host Key Document (HKD) certificate to the temporary directory by running the following command:

      $ cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/<hkd_cert.crt>

      Note

      The HKD certificate must be the same certificate that you downloaded when you created the peer pods secret.

   4. Obtain the attestation policy fields by running the se_parse_hdr.py script:

      $ python3 /tmp/se_parse_hdr.py /tmp/ibmse/hdr/hdr.bin /tmp/ibmse/hkds/<hkd_cert.crt>

      Example output

      ...
        ================================================
        se.image_phkh: xxx
        se.version: 256
        se.tag: xxx
        se.attestation_phkh: xxx

      Record these values for the SE attestation policy config map.

2. Obtain the certificates and certificate revocation lists (CRLs) by performing the following steps:

   1. Create a temporary directory for certificates by running the following command:

      $ mkdir /tmp/ibmse/certs

   2. Download the ibm-z-host-key-signing-gen2.crt certificate by running the following command:

      $ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt

   3. Download the DigiCertCA.crt certificate by running the following command:

      $ wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt

   4. Create a temporary directory for the CRLs by running the following command:

      $ mkdir /tmp/ibmse/crls

   5. Download the DigiCertTrustedRootG4.crl file by running the following command:

      $ wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl

   6. Download the DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl file by running the following command:

      $ wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl

3. Generate the RSA keys:

   1. Generate an RSA key pair by running the following command:

      $ openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 

      1

      1

      Specify the RSA key password.

   2. Create a temporary directory for the RSA keys by running the following command:

      $ mkdir /tmp/ibmse/rsa

   3. Create an encrypt_key.pub key by running the following command:

      $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub

   4. Create an encrypt_key.pem key by running the following command:

      $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem

4. Verify the structure of the /tmp/ibmse directory by running the following command:

   $ tree /tmp/ibmse

   Example output

   /tmp/ibmse
   ├── certs
   │   ├── ibm-z-host-key-signing-gen2.crt
   |   └── DigiCertCA.crt
   ├── crls
   │   └── ibm-z-host-key-gen2.crl
   │   └── DigiCertTrustedRootG4.crl
   │   └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
   ├── hdr
   │   └── hdr.bin
   ├── hkds
   │   └── <hkd_cert.crt>
   └── rsa
       ├── encrypt_key.pem
       └── encrypt_key.pub

5. Copy these files to the OpenShift Container Platform worker nodes by performing the following steps:

   1. Create a compressed file from the /tmp/ibmse directory by running the following command:

      $ tar -czf ibmse.tar.gz -C /tmp/ibmse

   2. Copy the .tar.gz file to the bastion node in your cluster by running the following command:

      $ scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 

      1

      1

      Specify the IP address of the bastion node.

   3. Connect to the bastion node over SSH by running the following command:

      $ ssh root@<ocp_bastion_ip>

   4. Copy the .tar.gz file to each worker node by running the following command:

      $ scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 

      1

      1

      Specify the IP address of the worker node.

   5. Extract the .tar.gz on each worker node by running the following command:

      $ ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/'

   6. Update the ibmse folder permissions by running the following command:

      $ ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/'

Procedure

1. Create a persistent-volume.yaml manifest file:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: ibmse-pv
     namespace: trustee-operator-system
   spec:
     capacity:
       storage: 100Mi
     accessModes:
       - ReadOnlyMany
     storageClassName: ""
     local:
       path: /opt/confidential-containers/ibmse
     nodeAffinity:
       required:
         nodeSelectorTerms:
           - matchExpressions:
               - key: node-role.kubernetes.io/worker
                 operator: Exists

2. Create the persistent volume by running the following command:

   $ oc apply -f persistent-volume.yaml

3. Create a persistent-volume-claim.yaml manifest file:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ibmse-pvc
     namespace: trustee-operator-system
   spec:
     accessModes:
       - ReadOnlyMany
     storageClassName: ""
     resources:
       requests:
         storage: 100Mi

4. Create the persistent volume claim by running the following command:

   $ oc apply -f persistent-volume-claim.yaml

Procedure

1. Create an rvps-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: rvps-reference-values
     namespace: trustee-operator-system
   data:
     reference-values.json: |
       [ 

   1

   
       ]

   1

   Leave this value empty.

2. Create the RVPS config map by running the following command:

   $ oc apply -f rvps-configmap.yaml

3. Create one or more secrets to share with attested clients according to the following example:

   $ oc create secret generic kbsres1 --from-literal key1=<res1val1> \
     --from-literal key2=<res1val2> -n trustee-operator-system

   In this example, the kbsres1 secret has two entries (key1, key2), which the Trustee clients retrieve. You can add more secrets according to your requirements.

4. Create a resourcepolicy-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: resource-policy
     namespace: trustee-operator-system
   data:
     policy.rego: | 

   1

   
       package policy 

   2

   
       path := split(data["resource-path"], "/")
       default allow = false
       allow {
         count(path) == 3
         input["tee"] == "se"
       }

   1

   The name of the resource policy, policy.rego, must match the resource policy defined in the Trustee config map.

   2

   The resource policy follows the Open Policy Agent specification. This example allows the retrieval of all resources when the TEE is not the sample attester.

5. Create the resource policy config map by running the following command:

   $ oc apply -f resourcepolicy-configmap.yaml

6. Create an attestation-policy.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: attestation-policy
     namespace: trustee-operator-system
   data:
     default.rego: | 

   1

   
       package policy
       import rego.v1
       default allow = false
       converted_version := sprintf("%v", [input["se.version"]])
       allow if {
           input["se.attestation_phkh"] == "<se.attestation_phkh>" 

   2

   
           input["se.image_phkh"] == "<se.image_phkh>"
           input["se.tag"] == "<se.tag>"
           converted_version == "256"
       }

   1

   Do not modify the policy name.

   2

   Specify the attestation policy fields you obtained by running the se_parse_hdr.py script.

7. Create the attestation policy config map by running the following command:

   $ oc apply -f attestation-policy.yaml

Procedure

1. Create a kbsconfig-cr.yaml manifest file:

   apiVersion: confidentialcontainers.org/v1alpha1
   kind: KbsConfig
   metadata:
     labels:
       app.kubernetes.io/name: kbsconfig
       app.kubernetes.io/instance: kbsconfig
       app.kubernetes.io/part-of: trustee-operator
       app.kubernetes.io/managed-by: kustomize
       app.kubernetes.io/created-by: trustee-operator
     name: kbsconfig
     namespace: trustee-operator-system
   spec:
     kbsConfigMapName: kbs-config-cm
     kbsAuthSecretName: kbs-auth-public-key
     kbsDeploymentType: AllInOneDeployment
     kbsRvpsRefValuesConfigMapName: rvps-reference-values
     kbsSecretResources: ["kbsres1"]
     kbsResourcePolicyConfigMapName: resource-policy
     kbsAttestationPolicyConfigMapName: attestation-policy
     kbsServiceType: NodePort
     ibmSEConfigSpec:
       certStorePvc: ibmse-pvc

2. Create the KbsConfig CR by running the following command:

   $ oc apply -f kbsconfig-cr.yaml

Verification

1. Set the default project by running the following command:

   $ oc project trustee-operator-system

2. Check the pods by running the following command:

   $ oc get pods -n trustee-operator-system

   Example output

   NAME                                                   READY   STATUS    RESTARTS   AGE
   trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
   trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

3. Set the POD_NAME environmental variable by running the following command:

   $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

4. Check the pod logs by running the following command:

   $ oc logs -n trustee-operator-system $POD_NAME

   Example output

   [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
   [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
   [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
   [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
   [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
   [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime

5. Expose the ibmse-pvc persistent volume claim to the Trustee pods by running the following command:

   $ oc patch deployment trustee-deployment --namespace=trustee-operator-system --type=json -p='[{"op": "remove", "path": "/spec/template/spec/volumes/5/persistentVolumeClaim/readOnly"}]'

6. Verify that the kbs-service is exposed on a node port by running the following command:

   $ oc get svc kbs-service -n trustee-operator-system

   Example output

   NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
   kbs-service   NodePort   198.51.100.54   <none>        8080:31862/TCP   23h

   The kbs-service URL is https://<worker_node_ip>:<node_port>, for example, https://172.16.0.56:31862.

Procedure

1. Create a verification-pod.yaml manifest file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: kbs-client
   spec:
     containers:
     - name: kbs-client
       image: quay.io/confidential-containers/kbs-client:latest
       imagePullPolicy: IfNotPresent
       command:
         - sleep
         - "360000"
       env:
         - name: RUST_LOG
           value:  none

2. Create the pod by running the following command:

   $ oc create -f verification-pod.yaml

3. Copy the https.crt file to the kbs-client pod by running the following command:

   $ oc cp https.crt kbs-client:/

4. Fetch the pod secret by running the following command:

   $ oc exec -it kbs-client -- kbs-client --cert-file https.crt \
     --url https://kbs-service:8080 get-resource \
     --path default/kbsres1/key1

   Example output

   res1val1

   The Trustee server returns the secret only if the attestation is successful.