Chapter 3. Deploying Red Hat build of Trustee for workloads running on IBM Z and IBM LinuxONE

Procedure

1. Create a trustee-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: trustee-operator-system

2. Create the trustee-operator-system namespace by running the following command:

   $ oc create -f trustee-namespace.yaml

3. Create a trustee-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: trustee-operator-group
     namespace: trustee-operator-system
   spec:
     targetNamespaces:
     - trustee-operator-system

4. Create the operator group by running the following command:

   $ oc create -f trustee-operatorgroup.yaml

5. Create a trustee-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: trustee-operator-system
     namespace: trustee-operator-system
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: trustee-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

6. Create the subscription by running the following command:

   $ oc create -f trustee-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n trustee-operator-system

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n trustee-operator-system

   Example output

   NAME                      DISPLAY                        PHASE
   trustee-operator.v0.4.2   Trustee Operator  0.4.2        Succeeded

Procedure

1. Create a kbs-config-cm.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kbs-config-cm
     namespace: trustee-operator-system
   data:
     kbs-config.toml: |
       [http_server]
       sockets = ["0.0.0.0:8080"]
       insecure_http = false
       private_key = "/etc/https-key/https.key"
       certificate = "/etc/https-cert/https.crt"
   
       [admin]
       insecure_api = false
       auth_public_key = "/etc/auth-secret/publicKey"
   
       [attestation_token]
       insecure_key = true
       attestation_token_type = "CoCo"
   
       [attestation_service]
       type = "coco_as_builtin"
       work_dir = "/opt/confidential-containers/attestation-service"
       policy_engine = "opa"
   
       [attestation_service.attestation_token_broker]
       type = "Simple"
       policy_dir = "/opt/confidential-containers/attestation-service/policies"
   
       [attestation_service.attestation_token_config]
       duration_min = 5
   
       [attestation_service.rvps_config]
       type = "BuiltIn"
   
       [attestation_service.rvps_config.storage]
       type = "LocalJson"
       file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"
   
       [[plugins]]
       name = "resource"
       type = "LocalFs"
       dir_path = "/opt/confidential-containers/kbs/repository"
   
       [policy_engine]
       policy_path = "/opt/confidential-containers/opa/policy.rego"

2. Create the config map by running the following command:

   $ oc create -f kbs-config-cm.yaml

Procedure

1. Create an rvps-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: rvps-reference-values
     namespace: trustee-operator-system
   data:
     reference-values.json: |
       [
       ]

2. Create the RVPS config map by running the following command:

   $ oc create -f rvps-configmap.yaml

Procedure

1. Generate the Key Broker Service (KBS) certificate and key by performing the following steps:

   1. Create the kbs.conf configuration file according to the following example:

      [req]
      default_bits = 2048
      default_keyfile = localhost.key
      distinguished_name = req_distinguished_name
      req_extensions = req_ext
      x509_extensions = v3_ca
      
      [req_distinguished_name]
      countryName = Country Name (2-letter code)
      countryName_default = <country_name>
      stateOrProvinceName = State or Province Name (full name)
      stateOrProvinceName_default = <state_name>
      localityName = Locality Name (eg, city)
      localityName_default = <locality_name>
      organizationName = Organization Name (eg, company)
      organizationName_default = Red Hat
      organizationalUnitName = organizationalunit
      organizationalUnitName_default = Development
      commonName = Common Name (e.g. server FQDN or
      YOUR name)
      commonName_default = kbs-service
      commonName_max = 64
      
      [req_ext]
      subjectAltName = @alt_names
      
      [v3_ca]
      subjectAltName = @alt_names
      
      [alt_names]
      IP.1  = <trustee_ip>
      DNS.1  = localhost
      DNS.2  = 127.0.0.1

   2. Generate the KBS key and self-signed certificate by running the following command:

      openssl req -x509 -nodes -days 365 \
         -newkey rsa:2048 \
         -keyout kbs.key \
         -out kbs.crt \
         -config kbs.conf \
         -passin pass:

   3. Copy the KBS key to the ibmse directory by running the following command:

      $ cp kbs.key /tmp/ibmse/kbs.key

   4. Copy the KBS certificate to the ibmse directory by running the following command:

      $ cp kbs.crt /tmp/ibmse/kbs.crt

2. Obtain the attestation policy fields by performing the following steps:

   1. Create a directory to download the GetRvps.sh script by running the following command:

      $ mkdir -p Rvps-Extraction/

   2. Download the script by running the following command:

      $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/GetRvps.sh -O $PWD/GetRvps.sh

   3. Create a subdirectory by running the following command:

      $ mkdir -p Rvps-Extraction/static-files

   4. Go to the static-files directory by running the following command:

      $ cd Rvps-Extraction/static-files

   5. Download the pvextract-hdr tool by running the following command:

      $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/pvextract-hdr -O $PWD/pvextract-hdr

   6. Make the tool executable by running the following command:

      $ chmod +x pvextract-hdr

   7. Download the se_parse_hdr.py script by running the following command:

      $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/se_parse_hdr.py -O $PWD/se_parse_hdr.py

   8. Copy your Host Key Document (HKD) certificate to the static-files directory by running the following command:

      $ cp ~/path/to/<hkd_cert.crt> .

      The static-files directory contains the following files:

      • HKD.crt
      • pvextract-hdr
      • se_parse_hdr.py

   9. Go to the Rvps-Extraction directory by running the following command:

      $ cd ..

   10. Make the GetRvps.sh script executable by running the following command:

       $ chmod +x GetRvps.sh

   11. Run the script:

       $ ./GetRvps.sh

       Example output

       ***Installing necessary packages for RVPS values extraction ***
       Updating Subscription Management repositories.
       Last metadata expiration check: 0:37:12 ago on Mon Nov 18 09:20:29 2024.
       Package python3-3.9.19-8.el9_5.1.s390x is already installed.
       Package python3-cryptography-36.0.1-4.el9.s390x is already installed.
       Package kmod-28-10.el9.s390x is already installed.
       Dependencies resolved.
       Nothing to do.
       Complete!
       ***Installation Finished ***
       1) Generate the RVPS From Local Image from User pc
       2) Generate RVPS from Volume
       3) Quit
       Please enter your choice:

       1. Enter 2 to generate the Reference Value Provider Service from the volume:

          Please enter your choice: 2

       2. Enter fa-pp for the libvirt pool name:

          Enter the Libvirt Pool Name: fa-pp

       3. Enter the libvirt gateway URI:

          Enter the Libvirt URI Name: <libvirt-uri> 

          1

          1

          Specify the LIBVIRT_URI value that you used to create the peer pods secret.

       4. Enter fa-pp-vol for the libvirt volume name:

          Enter the Libvirt Volume Name: fa-pp-vol

          Example output

          Downloading from PODVM Volume...
          
          mount: /mnt/myvm: special device /dev/nbd3p1 does not exist.
          Error: Failed to mount the image. Retrying...
          Mounting on second attempt passed
          /dev/nbd3 disconnected
          SE header found at offset 0x014000
          SE header written to '/root/Rvps-Extraction/output-files/hdr.bin' (640 bytes)
          se.tag:  42f3fe61e8a7e859cab3bb033fd11c61
          se.image_phkh:  92d0aff6eb86719b6b1ea0cb98d2c99ff2ec693df3efff2158f54112f6961508
          provenance = ewogICAgInNlLmF0dGVzdGF0aW9uX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnRhZyI6IFsKICAgICAgICAiNDJmM2ZlNjFlOGE3ZTg1OWNhYjNiYjAzM2ZkMTFjNjEiCiAgICBdLAogICAgInNlLmltYWdlX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnVzZXJfZGF0YSI6IFsKICAgICAgICAiMDAiCiAgICBdLAogICAgInNlLnZlcnNpb24iOiBbCiAgICAgICAgIjI1NiIKICAgIF0KfQo=
          -rw-r--r--. 1 root root 640 Dec 16 10:57 /root/Rvps-Extraction/output-files/hdr.bin
          -rw-r--r--. 1 root root 446 Dec 16 10:57 /root/Rvps-Extraction/output-files/ibmse-policy.rego
          -rw-r--r--. 1 root root 561 Dec 16 10:57 /root/Rvps-Extraction/output-files/se-message

3. Obtain the certificates and certificate revocation lists (CRLs) by performing the following steps:

   1. Create a temporary directory for certificates by running the following command:

      $ mkdir /tmp/ibmse/certs

   2. Download the ibm-z-host-key-signing-gen2.crt certificate by running the following command:

      $ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt

   3. Download the DigiCertCA.crt certificate by running the following command:

      $ wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt

   4. Create a temporary directory for the CRLs by running the following command:

      $ mkdir /tmp/ibmse/crls

   5. Download the ibm-z-host-key-gen2.crl file by running the following command:

      $ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-gen2.crl -O /tmp/ibmse/crls/ibm-z-host-key-gen2.crl

   6. Download the DigiCertTrustedRootG4.crl file by running the following command:

      $ wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl

   7. Download the DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl file by running the following command:

      $ wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl

   8. Create a temporary directory for the hdr.bin file by running the following command:

      $ mkdir -p /tmp/ibmse/hdr/

   9. Copy the hdr.bin file to the hdr directory by running the following command:

      $ cp /root/Rvps-Extraction/output-files/hdr.bin /tmp/ibmse/hdr/

   10. Create a temporary directory for Host Key Document (HKD) certificate by running the following command:

       $ mkdir -p /tmp/ibmse/hkds

   11. Copy your HKD certificate to the hkds directory by running the following command:

       $ cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/

4. Generate the RSA keys:

   1. Generate an RSA key pair by running the following command:

      $ openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 

      1

      1

      Specify the RSA key password.

   2. Create a temporary directory for the RSA keys by running the following command:

      $ mkdir -p /tmp/ibmse/rsa

   3. Create an encrypt_key.pub key by running the following command:

      $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub

   4. Create an encrypt_key.pem key by running the following command:

      $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem

5. Verify the structure of the /tmp/ibmse directory by running the following command:

   $ tree /tmp/ibmse

   Example output

   /tmp/ibmse
   ├──kbs.key
   ├──kbs.crt
   |
   ├── certs
   │ ├── ibm-z-host-key-signing-gen2.crt
   | └── DigiCertCA.crt
   ├── crls
   │ └── ibm-z-host-key-gen2.crl
   │ └── DigiCertTrustedRootG4.crl
   │ └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
   ├── hdr
   │ └── hdr.bin
   ├── hkds
   │ └── <hkd_cert.crt>
   └── rsa
   ├── encrypt_key.pem
   └── encrypt_key.pub

6. Copy these files to the OpenShift Container Platform worker nodes by performing the following steps:

   1. Create a compressed file from the /tmp/ibmse directory by running the following command:

      $ tar -czf ibmse.tar.gz -C /tmp/ ibmse

   2. Copy the .tar.gz file to the bastion node in your cluster by running the following command:

      $ scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 

      1

      1

      Specify the IP address of the bastion node.

   3. Connect to the bastion node over SSH by running the following command:

      $ ssh root@<ocp_bastion_ip>

   4. Copy the .tar.gz file to each worker node by running the following command:

      $ scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 

      1

      1

      Specify the IP address of the worker node.

   5. Extract the .tar.gz on each worker node by running the following command:

      $ ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/'

   6. Update the ibmse folder permissions by running the following command:

      $ ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/'

7. Create the secrets in the cluster with the KBS key and certificate by performing the following steps:

   1. Create a kbs-https-certificate.yaml manifest file according to the following example:

      apiVersion: v1
      kind: Secret
      metadata:
       name: kbs-https-certificate
       namespace: trustee-operator-system
      data:
       https.crt: $(cat /tmp/ibmse/kbs.crt | base64 -w 0)

   2. Create the secret with the KBS certificate by running the following command:

      $ oc create -f kbs-https-certificate.yaml

   3. Create a kbs-https-key.yaml manifest file according to the following example:

      apiVersion: v1
      kind: Secret
      metadata:
        name: kbs-https-key
        namespace: trustee-operator-system
      data:
       https.key: $(cat /tmp/ibmse/kbs.key | base64 -w 0)

   4. Create the secret with the KBS key by running the following command:

      $ oc create -f kbs-https-key.yaml

Procedure

1. Create a persistent-volume.yaml manifest file:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: ibmse-pv
     namespace: trustee-operator-system
   spec:
     capacity:
       storage: 100Mi
     accessModes:
       - ReadOnlyMany
     storageClassName: ""
     local:
       path: /opt/confidential-containers/ibmse
     nodeAffinity:
       required:
         nodeSelectorTerms:
           - matchExpressions:
               - key: node-role.kubernetes.io/worker
                 operator: Exists

2. Create the persistent volume by running the following command:

   $ oc create -f persistent-volume.yaml

3. Create a persistent-volume-claim.yaml manifest file:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ibmse-pvc
     namespace: trustee-operator-system
   spec:
     accessModes:
       - ReadOnlyMany
     storageClassName: ""
     resources:
       requests:
         storage: 100Mi

4. Create the persistent volume claim by running the following command:

   $ oc create -f persistent-volume-claim.yaml

Procedure

1. Create an attestation-policy.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: attestation-policy
     namespace: trustee-operator-system
   data:
     default.rego: |
       package policy
       import rego.v1
       default allow = false
       converted_version := sprintf("%v", [input["se.version"]])
       allow if {
           input["se.attestation_phkh"] == "<se.attestation_phkh>"
           input["se.image_phkh"] == "<se.image_phkh>"
           input["se.tag"] == "<se.tag>"
           converted_version == "256"
       }

   default.rego

   Do not modify the policy name.

   <se.attestation_phkh>

   Replace this with the attestation policy fields you obtained by running the se_parse_hdr.py script.

2. Create the attestation policy config map by running the following command:

   $ oc create -f attestation-policy.yaml

Procedure

1. Create a secret for container image signature verification by running the following command:

   $ oc create secret generic <type> \ 

   1

   
     --from-file=<tag>=./<public_key_file> \ 

   2

   
     -n trustee-operator-system

   1

   Specify the KBS secret type, for example, img-sig.

   2

   Specify the secret tag, for example, pub-key, and the public container image signing key.

2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

Procedure

1. Create a security-policy-config.json file according to the following examples:

   • Without signature verification:

     {
       "default": [
       {
         "type": "insecureAcceptAnything"
       }],
       "transports": {}
     }

   • With signature verification:

     {
         "default": [
         ],
         "transports": {
             "docker": {
                 "<container_registry_url>/<username>/busybox:latest": 

     1

     
                 [
                     {
                         "type": "sigstoreSigned",
                         "keyPath": "kbs:///default/img-sig/pub-key" 

     2

     
                     }
                 ]
             }
         }
     }

     1

     Specify the container registry URL, for example, "quay.io".

     2

     Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.

2. Create the security policy by running the following command:

   $ oc create secret generic security-policy \
     --from-file=osc=./<security-policy-config.json> \
     -n trustee-operator-system

   Do not alter the secret type, security-policy, or the key, osc.

   The security-policy secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

Procedure

1. Create a resourcepolicy-configmap.yaml manifest file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: resource-policy
     namespace: trustee-operator-system
   data:
     policy.rego: |
       package policy
       default allow = false
       allow {
         input["tee"] == "se"
       }

   policy.rego

   The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.

   package policy

   The resource policy follows the Open Policy Agent specification.

2. Create the resource policy config map by running the following command:

   $ oc create -f resourcepolicy-configmap.yaml

Procedure

1. Create an edge route by running the following command:

   $ oc create route edge --service=kbs-service --port kbs-port \
     -n trustee-operator-system

   Note

   Currently, only a route with a valid CA-signed certificate is supported. You cannot use a route with a self-signed certificate.

2. Set the TRUSTEE_HOST variable by running the following command:

   $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
     -o jsonpath={.spec.host})

3. Verify the route by running the following command:

   $ echo $TRUSTEE_HOST

   Example output

   kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io

Procedure

1. Create a private key by running the following command:

   $ openssl genpkey -algorithm ed25519 > privateKey

2. Create a public key by running the following command:

   $ openssl pkey -in privateKey -pubout -out publicKey

3. Create a secret by running the following command:

   $ oc create secret generic kbs-auth-public-key \
     --from-file=publicKey -n trustee-operator-system

4. Verify the secret by running the following command:

   $ oc get secret -n trustee-operator-system

Procedure

1. Create a kbsconfig-cr.yaml manifest file:

   apiVersion: confidentialcontainers.org/v1alpha1
   kind: KbsConfig
   metadata:
     labels:
       app.kubernetes.io/name: kbsconfig
       app.kubernetes.io/instance: kbsconfig
       app.kubernetes.io/part-of: trustee-operator
       app.kubernetes.io/managed-by: kustomize
       app.kubernetes.io/created-by: trustee-operator
     name: kbsconfig
     namespace: trustee-operator-system
   spec:
     kbsConfigMapName: kbs-config-cm
     kbsAuthSecretName: kbs-auth-public-key
     kbsDeploymentType: AllInOneDeployment
     kbsRvpsRefValuesConfigMapName: rvps-reference-values
     kbsSecretResources:
     - kbsres1
     - security-policy
     - <type>
     kbsResourcePolicyConfigMapName: resource-policy
     kbsAttestationPolicyConfigMapName: attestation-policy
     kbsHttpsKeySecretName: kbs-https-key
     kbsHttpsCertSecretName: kbs-https-certificate
     kbsServiceType: NodePort
     ibmSEConfigSpec:
       certStorePvc: ibmse-pvc
     KbsEnvVars:
       SE_SKIP_CERTS_VERIFICATION: "false"

   • kbsSecretResources. Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
   • SE_SKIP_CERTS_VERIFICATION: Set to true only for testing purposes.

2. Create the KbsConfig CR by running the following command:

   $ oc create -f kbsconfig-cr.yaml

Procedure

1. Set the default project by running the following command:

   $ oc project trustee-operator-system

2. Check the pods by running the following command:

   $ oc get pods -n trustee-operator-system

   Example output

   NAME                                                   READY   STATUS    RESTARTS   AGE
   trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
   trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

3. Set the POD_NAME environmental variable by running the following command:

   $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

4. Check the pod logs by running the following command:

   $ oc logs -n trustee-operator-system $POD_NAME

   Example output

   [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
   [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
   [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
   [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
   [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
   [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime