Deploying Red Hat build of Trustee


OpenShift sandboxed containers 1.11

Secure management and attestation of confidential containers workloads

Red Hat Customer Content Services

Abstract

Red Hat build of Trustee provides secure management and attestation of confidential containers workloads. You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment. Then, you deploy confidential containers on a separate cluster for your workload.

Preface

Chapter 1. Overview

Learn about Red Hat build of Trustee features and terminology. You must ensure that your OpenShift Container Platform environment is compatible.

1.1. About Red Hat build of Trustee

Red Hat build of Trustee is a critical component of the confidential containers solution for OpenShift sandboxed containers. Red Hat build of Trustee enables secure management and attestation of confidential workloads running within Trusted Execution Environments (TEEs) on a Red Hat OpenShift Container Platform cluster.

Red Hat build of Trustee is a service that facilitates the deployment and management of confidential containers. It provides attestation and key management services to ensure the integrity and confidentiality of workloads running in TEEs.

You deploy Red Hat build of Trustee on a separate OpenShift Container Platform cluster in a trusted environment, not on the cluster that hosts the primary workload. This separation enhances security by isolating sensitive operations from the untrusted cloud infrastructure.

Red Hat build of Trustee performs the following key functions:

  • Verifies the integrity of the TEE and the workloads running within it, ensuring that it only executes trusted code and data.
  • Securely manages cryptographic keys and secrets required by confidential containers, protecting sensitive data from unauthorized access.
  • Simplifies the configuration of TEEs through the KbsConfig custom resource, enabling seamless integration with OpenShift Container Platform workflows.

By leveraging Red Hat build of Trustee, OpenShift Container Platform users can deploy confidential workloads using familiar tools while maintaining strong security guarantees, even on shared or third-party infrastructure.

1.2. Providing feedback on Red Hat documentation

You can provide feedback or report an error by submitting the Create Issue form in Jira:

  1. Ensure that you are logged in to Jira. If you do not have a Jira account, you must create a Red Hat Jira account.
  2. Launch the Create Issue form.
  3. Complete the Summary, Description, and Reporter fields.

    In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue.

  4. Click Create.

You can deploy Red Hat build of Trustee for confidential containers workloads running on bare metal servers. bare metal servers in a disconnected network environment.

Optionally, you can deploy Red Hat build of Trustee with a cluster-wide proxy. For details, see Configuring the cluster-wide proxy in the OpenShift Container Platform documentation.

2.1. Prerequisites

2.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

  1. Install the Red Hat build of Trustee Operator.
  2. Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
  3. Create the Reference Value Provider Service (RVPS) config map.
  4. Create the attestation policy config map.
  5. Optional: Create a config map for Intel® TDX.
  6. Optional: Create a secret for custom keys clients.
  7. Optional: Create a secret for container image signature verification.
  8. Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
  9. Create the resource policy config map.
  10. Create the KBSConfig CR.
  11. Create the cluster route.
  12. Create the authentication secret.
  13. Verify the Red Hat build of Trustee configuration.

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI tool (oc).

Procedure

  1. Create a trustee-namespace.yaml manifest file:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Create the trustee-operator-system namespace by running the following command:

    $ oc create -f trustee-namespace.yaml
    Copy to Clipboard Toggle word wrap
  3. Create a trustee-operatorgroup.yaml manifest file:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: trustee-operator-group
      namespace: trustee-operator-system
    spec:
      targetNamespaces:
      - trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Create the operator group by running the following command:

    $ oc create -f trustee-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap
  5. Create a trustee-subscription.yaml manifest file:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: trustee-operator-system
      namespace: trustee-operator-system
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: trustee-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    Copy to Clipboard Toggle word wrap
  6. Create the subscription by running the following command:

    $ oc create -f trustee-subscription.yaml
    Copy to Clipboard Toggle word wrap
  7. Verify that the Operator is correctly installed by running the following command:

    $ oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command:

    $ watch oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                      DISPLAY                        PHASE
    trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded
    Copy to Clipboard Toggle word wrap

2.4. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

  1. Create a kbs-config-cm.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: kbs-config-cm
      namespace: trustee-operator-system
    data:
      kbs-config.toml: |
        [http_server]
        sockets = ["0.0.0.0:8080"]
        insecure_http = false
        private_key = "/etc/https-key/tls.key"
        certificate = "/etc/https-cert/tls.crt"
        worker_count = 4
    
        [admin]
        insecure_api = false
    
        auth_public_key = "/etc/auth-secret/publicKey"
    
        [attestation_token]
        attestation_token_type = "CoCo"
    
        [attestation_service]
        type = "coco_as_builtin"
        work_dir = "/opt/confidential-containers/attestation-service"
        policy_engine = "opa"
    
        [attestation_service.attestation_token_broker]
        policy_dir = "/opt/confidential-containers/attestation-service/policies"
    
    
        [attestation_service.attestation_token_config]
        duration_min = 5
    
        [attestation_service.rvps_config]
        type = "BuiltIn"
    
        [attestation_service.rvps_config.storage]
        type = "LocalJson"
        file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"
    
        [[plugins]]
        name = "resource"
        type = "LocalFs"
        dir_path = "/opt/confidential-containers/kbs/repository"
    
        [policy_engine]
        policy_path = "/opt/confidential-containers/opa/policy.rego"
    Copy to Clipboard Toggle word wrap
  2. Create the config map by running the following command:

    $ oc create -f kbs-config-cm.yaml
    Copy to Clipboard Toggle word wrap

2.5. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

  1. Create an rvps-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rvps-reference-values
      namespace: trustee-operator-system
    data:
      reference-values.json: |
        [
        ]
    Copy to Clipboard Toggle word wrap
  2. Create the RVPS config map by running the following command:

    $ oc create -f rvps-configmap.yaml
    Copy to Clipboard Toggle word wrap

2.6. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Open Policy Agent specification.

This policy checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 values against the reference values to ensure that the confidential containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. For details, see Linux TPM PCR Registry in the UAPI Group Specifications documentation.

Procedure

  1. Create an attestation-policy.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: attestation-policy
      namespace: trustee-operator-system
    data:
      default.rego: |
        package policy
    
        import rego.v1
        default executables := 33
        default hardware := 97
        default configuration := 36
    
        ##### Azure vTPM SNP
        executables := 3 if {
          input.azsnpvtpm.measurement in data.reference.measurement
          input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
        }
    
        hardware := 2 if {
          input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
          input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
          input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
          input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
        }
    
        configuration := 2 if {
          input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
          input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
          input.azsnpvtpm.policy_abi_major in data.reference.abi_major
          input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
          input.azsnpvtpm.policy_single_socket in data.reference.single_socket
          input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
        }
    
        ##### Azure vTPM TDX
        executables := 3 if {
          input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
          input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
          input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
          input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
          input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
        }
    
        hardware := 2 if {
          # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
          input.aztdxvtpm.quote.header.tee_type == "81000000"
          input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"
    
          # Check TDX Module version and its hash. Also check OVMF code hash.
          # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
          # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
          # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
        }
    
        configuration := 2 if {
          # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
        }
    Copy to Clipboard Toggle word wrap
  2. Create the attestation policy config map by running the following command:

    $ oc create -f attestation-policy.yaml
    Copy to Clipboard Toggle word wrap

2.7. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

  1. Create a tdx-config.yaml manifest file according to the following example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: tdx-config
      namespace: trustee-operator-system
    data:
      sgx_default_qcnl.conf: |
        {
          "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
        }
    Copy to Clipboard Toggle word wrap
  2. Create the tdx-config config map by running the following command:

    $ oc create -f tdx-config.yaml
    Copy to Clipboard Toggle word wrap

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

  • You have created one or more custom keys.

Procedure

  • Create a secret for the custom keys according to the following example:

    $ oc create secret generic attestation-status \
      --from-literal key1=<custom_key1> \ 
    1
    
      --from-literal key2=<custom_key2> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify a custom key.

    You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

  1. Create a secret for container image signature verification by running the following command:

    $ oc create secret generic <type> \ 
    1
    
      --from-file=<tag>=./<public_key_file> \ 
    2
    
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify the KBS secret type, for example, img-sig.
    2
    Specify the secret tag, for example, pub-key, and the public container image signing key.
  2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see containers-policy.json 5.

Note

Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.

Procedure

  1. Create a security-policy-config.json file according to the following example:

    {
      "default": [
          {
          "type": "reject" 
    1
    
          }
      ],
      "transports": {
          "<transport>": { 
    2
    
              "<registry>/<image>": 
    3
    
              [
                  {
                      "type": "sigstoreSigned",
                      "keyPath": "kbs:///default/<type>/<tag>" 
    4
    
                  }
              ]
          }
      }
    }
    Copy to Clipboard Toggle word wrap
    1
    By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
    2
    Specify the image repository for transport, for example, "docker":. For more information, see containers-transports 5.
    3
    Specify the container registry and image, for example, "quay.io/my-image".
    4
    Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
  2. Create the security policy by running the following command:

    $ oc create secret generic <security-policy-name> \
      --from-file=<osc-key>=./<security-policy-config.json> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

2.11. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

  1. Create a resourcepolicy-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: resource-policy
      namespace: trustee-operator-system
    data:
      policy.rego: |
        package policy
        default allow = true
        allow {
          input["submods"]["cpu0"]["ear.status"] == "affirming"
        }
    Copy to Clipboard Toggle word wrap
    policy.rego
    The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
    package policy
    The resource policy follows the Open Policy Agent specification.
  2. Create the resource policy config map by running the following command:

    $ oc create -f resourcepolicy-configmap.yaml
    Copy to Clipboard Toggle word wrap

2.12. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

  1. Create an edge route by running the following command:

    $ oc create route passthrough --service=kbs-service --port kbs-port \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Set the TRUSTEE_HOST variable by running the following command:

    $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
      -o jsonpath={.spec.host})
    Copy to Clipboard Toggle word wrap
  3. Verify the route by running the following command:

    $ echo $TRUSTEE_HOST
    Copy to Clipboard Toggle word wrap

    Example output

    kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
    Copy to Clipboard Toggle word wrap

2.13. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

  1. Create a private key by running the following command:

    $ openssl genpkey -algorithm ed25519 > privateKey
    Copy to Clipboard Toggle word wrap
  2. Create a public key by running the following command:

    $ openssl pkey -in privateKey -pubout -out publicKey
    Copy to Clipboard Toggle word wrap
  3. Create a secret by running the following command:

    $ oc create secret generic kbs-auth-public-key \
      --from-file=publicKey -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Verify the secret by running the following command:

    $ oc get secret -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

2.14. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

  1. Create a kbsconfig-cr.yaml manifest file:

    apiVersion: confidentialcontainers.org/v1alpha1
    kind: KbsConfig
    metadata:
      labels:
        app.kubernetes.io/name: kbsconfig
        app.kubernetes.io/instance: kbsconfig
        app.kubernetes.io/part-of: trustee-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: trustee-operator
      name: kbsconfig
      namespace: trustee-operator-system
    spec:
      kbsConfigMapName: kbs-config-cm
      kbsAuthSecretName: kbs-auth-public-key
      kbsDeploymentType: AllInOneDeployment
      kbsRvpsRefValuesConfigMapName: rvps-reference-values
      kbsSecretResources:
      - attestation-status
      - <security-policy-name>
      kbsResourcePolicyConfigMapName: resource-policy
      kbsHttpsKeySecretName: kbs-https-key
      kbsHttpsCertSecretName: kbs-https-certificate
      kbsAttestationCertSecretName: attestation-cert
      kbsAttestationKeySecretName: attestation-key
      kbsServiceType: NodePort
    # tdxConfigSpec:
    #   kbsTdxConfigMapName: tdx-config
    # kbsServiceType: <service_type>
    Copy to Clipboard Toggle word wrap
    • kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
    • Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
    • Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.
  2. Create the KbsConfig CR by running the following command:

    $ oc create -f kbsconfig-cr.yaml
    Copy to Clipboard Toggle word wrap

2.15. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

  1. Set the default project by running the following command:

    $ oc project trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Check the pods by running the following command:

    $ oc get pods -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                   READY   STATUS    RESTARTS   AGE
    trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
    trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m
    Copy to Clipboard Toggle word wrap

  3. Set the POD_NAME environmental variable by running the following command:

    $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
    Copy to Clipboard Toggle word wrap
  4. Check the pod logs by running the following command:

    $ oc logs -n trustee-operator-system $POD_NAME
    Copy to Clipboard Toggle word wrap

    Example output

    [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
    [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
    [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
    [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
    [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
    [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime
    Copy to Clipboard Toggle word wrap

You can deploy Red Hat build of Trustee for confidential containers workloads running on

2.16.1. Prerequisites

You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing a user-provisioned bare metal cluster on a disconnected environment

2.16.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

  1. Install the Red Hat build of Trustee Operator.
  2. Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
  3. Optional: Download VCEK certificates for AMD SEV-SNP nodes.
  4. Create the Reference Value Provider Service (RVPS) config map.
  5. Create the attestation policy config map.
  6. Optional: Create a config map for Intel® TDX.
  7. Optional: Create a secret for custom keys clients.
  8. Optional: Create a secret for container image signature verification.
  9. Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
  10. Create the resource policy config map.
  11. Create the KBSConfig CR.
  12. Create the cluster route.
  13. Create the authentication secret.
  14. Verify the Red Hat build of Trustee configuration.

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI tool (oc).
  • You have disabled the default catalog sources and mirrored the Operator catalog. For details, see Using Operator Lifecycle Manager in disconnected environments in the OpenShift Container Platform documentation.

Procedure

  1. Create a trustee-namespace.yaml manifest file:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Create the trustee-operator-system namespace by running the following command:

    $ oc create -f trustee-namespace.yaml
    Copy to Clipboard Toggle word wrap
  3. Create a trustee-operatorgroup.yaml manifest file:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: trustee-operator-group
      namespace: trustee-operator-system
    spec:
      targetNamespaces:
      - trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Create the operator group by running the following command:

    $ oc create -f trustee-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap
  5. Create a trustee-subscription.yaml manifest file:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: trustee-operator-system
      namespace: trustee-operator-system
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: trustee-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    Copy to Clipboard Toggle word wrap
  6. Create the subscription by running the following command:

    $ oc create -f trustee-subscription.yaml
    Copy to Clipboard Toggle word wrap
  7. Verify that the Operator is correctly installed by running the following command:

    $ oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command:

    $ watch oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                      DISPLAY                        PHASE
    trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded
    Copy to Clipboard Toggle word wrap

2.16.4. Creating HTTPS secrets

Generate keys to securely launch Red Hat build of Trustee and enables services to use HTTPS.

Procedure

  1. Set the DOMAIN variable for the cluster by running the following command:

    $ DOMAIN=$(oc get ingress.config/cluster -o jsonpath='{.spec.domain}')
    Copy to Clipboard Toggle word wrap
  2. Set the NS variable for the Red Hat build of Trustee namespace by running the following command:

    $ NS=trustee-operator-system
    Copy to Clipboard Toggle word wrap
  3. Set the ROUTE_NAME variable by running the following command:

    $ ROUTE_NAME=kbs-service
    Copy to Clipboard Toggle word wrap
  4. Set the ROUTE variable to the full DNS name by running the following command:

    $ ROUTE="${ROUTE_NAME}-${NS}.${DOMAIN}"
    Copy to Clipboard Toggle word wrap
  5. Generate a private SSL/TLS key and certificate for Red Hat build of Trustee by running the following command:

    $ openssl req -x509 -nodes -days 365 \
      -newkey rsa:2048 \
      -keyout tls.key \
      -out tls.crt \
      -subj "/CN=<custom_cn>/O=<custom_org>" \
      -addext "subjectAltName=DNS:${ROUTE}"
    Copy to Clipboard Toggle word wrap
    • <custom_cn> is a custom CN. For example: kbs-trustee-operator-system.
    • <custom_org> is a name of your organization.
  6. Create the kbs-https-certificate secret in the trustee-operator-system namespace by running the following command:

    $ oc create secret generic kbs-https-certificate --from-file=tls.crt -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  7. Create the kbs-https-key secret in the trustee-operator-system namespace by running the following command:

    $ oc create secret generic kbs-https-key --from-file=tls.key -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

2.16.5. Creating the attestation token secrets

Generate an attestation token key and certificate for Red Hat build of Trustee.

Procedure

  1. Generate a private elliptic curve SSL key called token.key by running the following command:

    $ openssl ecparam -name prime256v1 -genkey -noout -out token.key
    Copy to Clipboard Toggle word wrap
  2. Create the attestation-key secret from the SSL/TLS key in the trustee-operator-system namespace:

    $ oc create secret generic attestation-key \
      --from-file=token.key \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  3. Generate a self-signed SSL/TLS certificate from the private SSL key by running the following command:

    $ openssl req -new -x509 -key token.key -out token.crt -days 365 \
      -subj "/CN=<custom_cn>/O=<custom_org>"
    Copy to Clipboard Toggle word wrap
    • <custom_cn>: Specify the Common Name. For example: kbs-trustee-operator-system.
    • <custom_org>: Specify your organization name.
  4. Create the attestation-cert secret from the SSL/TLS key and certificate in the trustee-operator-system namespace:

    $ oc create secret generic attestation-cert \
      --from-file=token.crt \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  5. Create the attestation-status secret used for verifying the attestation process:

    $ oc create secret generic attestation-status \
      --from-literal=status=success \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

2.16.6. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

  1. Create a kbs-config-cm.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: kbs-config-cm
      namespace: trustee-operator-system
    data:
      kbs-config.toml: |
        [http_server]
        sockets = ["0.0.0.0:8080"]
        insecure_http = false
        private_key = "/etc/https-key/tls.key"
        certificate = "/etc/https-cert/tls.crt"
        worker_count = 4
    
        [admin]
        insecure_api = false
    
        auth_public_key = "/etc/auth-secret/publicKey"
    
        [attestation_token]
        insecure_key = false
        trusted_certs_paths = ["/etc/attestation-cert/token.crt"]
        attestation_token_type = "CoCo"
    
        [attestation_service]
        type = "coco_as_builtin"
        work_dir = "/opt/confidential-containers/attestation-service"
        policy_engine = "opa"
    
        [attestation_service.attestation_token_broker]
        type = "Ear"
        policy_dir = "/opt/confidential-containers/attestation-service/policies"
    
        [attestation_service.attestation_token_broker.signer]
        key_path = "/etc/attestation-key/token.key"
        cert_path = "/etc/attestation-cert/token.crt"
    
        [attestation_service.attestation_token_config]
        duration_min = 5
    
        [attestation_service.rvps_config]
        type = "BuiltIn"
    
        [attestation_service.rvps_config.storage]
        type = "LocalJson"
        file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"
    
        [[plugins]]
        name = "resource"
        type = "LocalFs"
        dir_path = "/opt/confidential-containers/kbs/repository"
    
        [policy_engine]
        policy_path = "/opt/confidential-containers/opa/policy.rego"
    Copy to Clipboard Toggle word wrap
  2. Create the config map by running the following command:

    $ oc create -f kbs-config-cm.yaml
    Copy to Clipboard Toggle word wrap

For each AMD SEV-SNP node, you must download a Versioned Chip Endorsement Key (VCEK) certificate so that Red Hat build of Trustee can sign the attestation report generated by the confidential containers virtual machine (VM).

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI tool (oc).
  • You have installed the podman utility.
  • You have installed the jq utility.

Procedure

  1. Perform the following steps on the worker node:

    1. Set the nodename variable by running the following command:

      $ export nodename=<node_name>
      Copy to Clipboard Toggle word wrap
    2. Retrieve the VCEK URL by running the following command:

      $ sudo podman run --privileged \
        quay.io/openshift_sandboxed_containers/coco-tools:latest \
        /tools/snphost show vcek-url \
        | cut -d '?' -f 1  > vcek_base_url_$nodename.txt
      Copy to Clipboard Toggle word wrap
    3. Obtain the VCEK URL by running the following command:

      $ VCEK_BASE_URL=$(cat vcek_base_url_$nodename.txt) \
      && echo $VCEK_BASE_URL
      Copy to Clipboard Toggle word wrap

      The command prints the full VCEK URL, including query parameters.

      Example output

      https://kdsintf.amd.com/vcek/v1/Genoa/126A06A956658EF040D5DF52D438BB78DBB8E8C1C6244B420513B00ECCF1E6E2456C925FD913E8F5A80C209F83052077871919E4F1EBB62B296B36E084B57095
      Copy to Clipboard Toggle word wrap

    4. Create a my-pod.yaml pod manifest according to the following example:

      apiVersion: v1
      kind: Pod
      metadata:
        name: coco-guest
        labels:
          app: coco-guest
        annotations:
          io.katacontainers.config.hypervisor.default_memory: "4096"
          io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
          io.katacontainers.config.hypervisor.cc_init_data: <data>
      spec:
        runtimeClassName: kata-cc
        nodeName: <node_name>
        containers:
          - name: snp-guest
            image: registry.access.redhat.com/ubi9/ubi:latest
            command:
              - sleep
              - "36000"
            securityContext:
              privileged: false
              seccompProfile:
                type: RuntimeDefault
      Copy to Clipboard Toggle word wrap
      <data>
      Base64-encoded initialization data used by the hypervisor to securely launch the confidential virtual machine (VM). For more information, see Creating initdata.
    5. Create the pod by running the following command:

      $ oc create pod -f my-pod.yaml
      Copy to Clipboard Toggle word wrap
    6. Obtain the pod attestation report by running the following command:

      $ oc exec -it coco-guest -- curl http://127.0.0.1:8006/aa/evidence?runtime_data=test > <attestation_report>-$nodename.txt
      Copy to Clipboard Toggle word wrap
      <attestation_report>

      Specify the attestation report file name.

      Example output

      {
        "attestation_report": {
          "version": 2,
          "guest_svn": 0,
          "policy": 196608,
          ...
          "reported_tcb": {
            "bootloader": 3,
            "tee": 0,
            "_reserved": [0, 0, 0, 0],
            "snp": 14,
            "microcode": 213
          },
         ...
        },
        "cert_chain": null
      }
      Copy to Clipboard Toggle word wrap

    7. Set the bootloader, tee, snp, and microcode values by running the following command:

      $ eval $(cat <attestation-report-name> | jq -r '
        .attestation_report.reported_tcb |
        "export bootloader=\(.bootloader);
         export tee=\(.tee);
         export snp=\(.snp);
         export microcode=\(.microcode);"
      ')
      
      echo "Bootloader: $bootloader"
      echo "TEE: $tee"
      echo "SNP: $snp"
      echo "Microcode: $microcode"
      Copy to Clipboard Toggle word wrap
    8. Set the VCEK_URL_NODE1 variable by running the following command:

      $ export VCEK_URL_NODE1="$VCEK_BASE_URL?blSPL=$bootloader&teeSPL=$tee&snpSPL=$snp&ucodeSPL=$microcode"
      Copy to Clipboard Toggle word wrap
    9. Create a separate directory for each processor type by running the following commands:

      $ mkdir -p ek/<processor_type>
      Copy to Clipboard Toggle word wrap
      <processor_type>
      Specify genoa, milan, or turin.
    10. Download certificates in separate directories for each processor type:

      $ curl -L -o ek/<processor_type>/vcek_node1.crt $VCEK_URL_NODE1
      Copy to Clipboard Toggle word wrap
      <processor_type>
      Specify genoa, milan, or turin.
  2. Create a secret for each processor from the VCEK certificates by running the following command:

    $ oc create secret generic <processor_type>-secret \
      --from-file ./ek/<processor_type> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    <processor_type>
    Specify genoa, milan, or turin.

2.16.8. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

  1. Create an rvps-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rvps-reference-values
      namespace: trustee-operator-system
    data:
      reference-values.json: |
        [
        ]
    Copy to Clipboard Toggle word wrap
  2. Create the RVPS config map by running the following command:

    $ oc create -f rvps-configmap.yaml
    Copy to Clipboard Toggle word wrap

2.16.9. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Open Policy Agent specification.

This policy checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 values against the reference values to ensure that the confidential containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. For details, see Linux TPM PCR Registry in the UAPI Group Specifications documentation.

Procedure

  1. Create an attestation-policy.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: attestation-policy
      namespace: trustee-operator-system
    data:
      default.rego: |
        package policy
    
        import rego.v1
        default executables := 33
        default hardware := 97
        default configuration := 36
    
        ##### Azure vTPM SNP
        executables := 3 if {
          input.azsnpvtpm.measurement in data.reference.measurement
          input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
        }
    
        hardware := 2 if {
          input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
          input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
          input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
          input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
        }
    
        configuration := 2 if {
          input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
          input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
          input.azsnpvtpm.policy_abi_major in data.reference.abi_major
          input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
          input.azsnpvtpm.policy_single_socket in data.reference.single_socket
          input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
        }
    
        ##### Azure vTPM TDX
        executables := 3 if {
          input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
          input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
          input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
          input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
          input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
        }
    
        hardware := 2 if {
          # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
          input.aztdxvtpm.quote.header.tee_type == "81000000"
          input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"
    
          # Check TDX Module version and its hash. Also check OVMF code hash.
          # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
          # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
          # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
        }
    
        configuration := 2 if {
          # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
        }
    Copy to Clipboard Toggle word wrap
  2. Create the attestation policy config map by running the following command:

    $ oc create -f attestation-policy.yaml
    Copy to Clipboard Toggle word wrap

2.16.10. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

  1. Create a tdx-config.yaml manifest file according to the following example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: tdx-config
      namespace: trustee-operator-system
    data:
      sgx_default_qcnl.conf: |
        {
          "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
        }
    Copy to Clipboard Toggle word wrap
  2. Create the tdx-config config map by running the following command:

    $ oc create -f tdx-config.yaml
    Copy to Clipboard Toggle word wrap

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

  • You have created one or more custom keys.

Procedure

  • Create a secret for the custom keys according to the following example:

    $ oc create secret generic attestation-status \
      --from-literal key1=<custom_key1> \ 
    1
    
      --from-literal key2=<custom_key2> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify a custom key.

    You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

  1. Create a secret for container image signature verification by running the following command:

    $ oc create secret generic <type> \ 
    1
    
      --from-file=<tag>=./<public_key_file> \ 
    2
    
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify the KBS secret type, for example, img-sig.
    2
    Specify the secret tag, for example, pub-key, and the public container image signing key.
  2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see containers-policy.json 5.

Note

Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.

Procedure

  1. Create a security-policy-config.json file according to the following example:

    {
      "default": [
          {
          "type": "reject" 
    1
    
          }
      ],
      "transports": {
          "<transport>": { 
    2
    
              "<registry>/<image>": 
    3
    
              [
                  {
                      "type": "sigstoreSigned",
                      "keyPath": "kbs:///default/<type>/<tag>" 
    4
    
                  }
              ]
          }
      }
    }
    Copy to Clipboard Toggle word wrap
    1
    By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
    2
    Specify the image repository for transport, for example, "docker":. For more information, see containers-transports 5.
    3
    Specify the container registry and image, for example, "quay.io/my-image".
    4
    Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
  2. Create the security policy by running the following command:

    $ oc create secret generic <security-policy-name> \
      --from-file=<osc-key>=./<security-policy-config.json> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

2.16.14. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

  1. Create a resourcepolicy-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: resource-policy
      namespace: trustee-operator-system
    data:
      policy.rego: |
        package policy
        default allow = true
        allow {
          input["submods"]["cpu0"]["ear.status"] == "affirming"
        }
    Copy to Clipboard Toggle word wrap
    policy.rego
    The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
    package policy
    The resource policy follows the Open Policy Agent specification.
  2. Create the resource policy config map by running the following command:

    $ oc create -f resourcepolicy-configmap.yaml
    Copy to Clipboard Toggle word wrap

2.16.15. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

  1. Create an edge route by running the following command:

    $ oc create route passthrough --service=kbs-service --port kbs-port \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Set the TRUSTEE_HOST variable by running the following command:

    $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
      -o jsonpath={.spec.host})
    Copy to Clipboard Toggle word wrap
  3. Verify the route by running the following command:

    $ echo $TRUSTEE_HOST
    Copy to Clipboard Toggle word wrap

    Example output

    kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
    Copy to Clipboard Toggle word wrap

2.16.16. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

  1. Create a private key by running the following command:

    $ openssl genpkey -algorithm ed25519 > privateKey
    Copy to Clipboard Toggle word wrap
  2. Create a public key by running the following command:

    $ openssl pkey -in privateKey -pubout -out publicKey
    Copy to Clipboard Toggle word wrap
  3. Create a secret by running the following command:

    $ oc create secret generic kbs-auth-public-key \
      --from-file=publicKey -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Verify the secret by running the following command:

    $ oc get secret -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

2.16.17. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

  1. Create a kbsconfig-cr.yaml manifest file:

    apiVersion: confidentialcontainers.org/v1alpha1
    kind: KbsConfig
    metadata:
      labels:
        app.kubernetes.io/name: kbsconfig
        app.kubernetes.io/instance: kbsconfig
        app.kubernetes.io/part-of: trustee-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: trustee-operator
      name: kbsconfig
      namespace: trustee-operator-system
    spec:
      kbsConfigMapName: kbs-config-cm
      kbsAuthSecretName: kbs-auth-public-key
      kbsDeploymentType: AllInOneDeployment
      kbsRvpsRefValuesConfigMapName: rvps-reference-values
      kbsSecretResources:
      - attestation-status
      - <security-policy-name>
      kbsResourcePolicyConfigMapName: resource-policy
      kbsHttpsKeySecretName: kbs-https-key
      kbsHttpsCertSecretName: kbs-https-certificate
      kbsAttestationCertSecretName: attestation-cert
      kbsAttestationKeySecretName: attestation-key
      kbsServiceType: NodePort
    # tdxConfigSpec:
    #   kbsTdxConfigMapName: tdx-config
    # kbsServiceType: <service_type>
      kbsLocalCertCacheSpec:
        secrets:
        - secretName: milan-secret
          mountPath: "/etc/kbs/snp/ek/milan"
        - secretName: genoa-secret
          mountPath: "/etc/kbs/snp/ek/genoa"
        - secretName: turin-secret
          mountPath: "/etc/kbs/snp/ek/turin"
    Copy to Clipboard Toggle word wrap
    • kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
    • Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
    • Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.
    • kbsLocalCertCacheSpec: For workloads running on AMD SEV-SNP nodes, specify the VCEK certificates in the kbsLocalCertCacheSpec stanza.
  2. Create the KbsConfig CR by running the following command:

    $ oc create -f kbsconfig-cr.yaml
    Copy to Clipboard Toggle word wrap

2.16.18. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

  1. Set the default project by running the following command:

    $ oc project trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Check the pods by running the following command:

    $ oc get pods -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                   READY   STATUS    RESTARTS   AGE
    trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
    trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m
    Copy to Clipboard Toggle word wrap

  3. Set the POD_NAME environmental variable by running the following command:

    $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
    Copy to Clipboard Toggle word wrap
  4. Check the pod logs by running the following command:

    $ oc logs -n trustee-operator-system $POD_NAME
    Copy to Clipboard Toggle word wrap

    Example output

    [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
    [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
    [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
    [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
    [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
    [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime
    Copy to Clipboard Toggle word wrap

You can deploy Red Hat build of Trustee for confidential containers workloads running on Azure Red Hat OpenShift.

3.1. Prerequisites

3.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

  1. Install the Red Hat build of Trustee Operator.
  2. Create HTTPS secrets.
  3. Create the attestation token secret.
  4. Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
  5. Create the Reference Value Provider Service (RVPS) config map. Initially, you create an empty config map for the reference values. You update the values after you create KBSConfig custom resource (CR).
  6. Create the attestation policy config map.
  7. Optional: Create a config map for Intel® TDX.
  8. Optional: Create a secret for custom keys clients.
  9. Optional: Create a secret for container image signature verification.
  10. Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
  11. Create the resource policy config map.
  12. Create the KBSConfig CR.
  13. Create the cluster route.
  14. Create the authentication secret.
  15. Update the RVPS config map with the reference values.
  16. Verify the Red Hat build of Trustee configuration.

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI tool (oc).

Procedure

  1. Create a trustee-namespace.yaml manifest file:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Create the trustee-operator-system namespace by running the following command:

    $ oc create -f trustee-namespace.yaml
    Copy to Clipboard Toggle word wrap
  3. Create a trustee-operatorgroup.yaml manifest file:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: trustee-operator-group
      namespace: trustee-operator-system
    spec:
      targetNamespaces:
      - trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Create the operator group by running the following command:

    $ oc create -f trustee-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap
  5. Create a trustee-subscription.yaml manifest file:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: trustee-operator-system
      namespace: trustee-operator-system
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: trustee-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    Copy to Clipboard Toggle word wrap
  6. Create the subscription by running the following command:

    $ oc create -f trustee-subscription.yaml
    Copy to Clipboard Toggle word wrap
  7. Verify that the Operator is correctly installed by running the following command:

    $ oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command:

    $ watch oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                      DISPLAY                        PHASE
    trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded
    Copy to Clipboard Toggle word wrap

3.4. Creating HTTPS secrets

Generate keys to securely launch Red Hat build of Trustee and enables services to use HTTPS.

Procedure

  1. Set the DOMAIN variable for the cluster by running the following command:

    $ DOMAIN=$(oc get ingress.config/cluster -o jsonpath='{.spec.domain}')
    Copy to Clipboard Toggle word wrap
  2. Set the NS variable for the Red Hat build of Trustee namespace by running the following command:

    $ NS=trustee-operator-system
    Copy to Clipboard Toggle word wrap
  3. Set the ROUTE_NAME variable by running the following command:

    $ ROUTE_NAME=kbs-service
    Copy to Clipboard Toggle word wrap
  4. Set the ROUTE variable to the full DNS name by running the following command:

    $ ROUTE="${ROUTE_NAME}-${NS}.${DOMAIN}"
    Copy to Clipboard Toggle word wrap
  5. Generate a private SSL/TLS key and certificate for Red Hat build of Trustee by running the following command:

    $ openssl req -x509 -nodes -days 365 \
      -newkey rsa:2048 \
      -keyout tls.key \
      -out tls.crt \
      -subj "/CN=<custom_cn>/O=<custom_org>" \
      -addext "subjectAltName=DNS:${ROUTE}"
    Copy to Clipboard Toggle word wrap
    • <custom_cn> is a custom CN. For example: kbs-trustee-operator-system.
    • <custom_org> is a name of your organization.
  6. Create the kbs-https-certificate secret in the trustee-operator-system namespace by running the following command:

    $ oc create secret generic kbs-https-certificate --from-file=tls.crt -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  7. Create the kbs-https-key secret in the trustee-operator-system namespace by running the following command:

    $ oc create secret generic kbs-https-key --from-file=tls.key -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

3.5. Creating the attestation token secrets

Generate an attestation token key and certificate for Red Hat build of Trustee.

Procedure

  1. Generate a private elliptic curve SSL key called token.key by running the following command:

    $ openssl ecparam -name prime256v1 -genkey -noout -out token.key
    Copy to Clipboard Toggle word wrap
  2. Create the attestation-key secret from the SSL/TLS key in the trustee-operator-system namespace:

    $ oc create secret generic attestation-key \
      --from-file=token.key \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  3. Generate a self-signed SSL/TLS certificate from the private SSL key by running the following command:

    $ openssl req -new -x509 -key token.key -out token.crt -days 365 \
      -subj "/CN=<custom_cn>/O=<custom_org>"
    Copy to Clipboard Toggle word wrap
    • <custom_cn>: Specify the Common Name. For example: kbs-trustee-operator-system.
    • <custom_org>: Specify your organization name.
  4. Create the attestation-cert secret from the SSL/TLS key and certificate in the trustee-operator-system namespace:

    $ oc create secret generic attestation-cert \
      --from-file=token.crt \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  5. Create the attestation-status secret used for verifying the attestation process:

    $ oc create secret generic attestation-status \
      --from-literal=status=success \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

3.6. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

  1. Create a kbs-config-cm.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: kbs-config-cm
      namespace: trustee-operator-system
    data:
      kbs-config.toml: |
        [http_server]
        sockets = ["0.0.0.0:8080"]
        insecure_http = false
        private_key = "/etc/https-key/tls.key"
        certificate = "/etc/https-cert/tls.crt"
        worker_count = 4
    
        [admin]
        insecure_api = false
    
        auth_public_key = "/etc/auth-secret/publicKey"
    
        [attestation_token]
        insecure_key = false
        trusted_certs_paths = ["/etc/attestation-cert/token.crt"]
        attestation_token_type = "CoCo"
    
        [attestation_service]
        type = "coco_as_builtin"
        work_dir = "/opt/confidential-containers/attestation-service"
        policy_engine = "opa"
    
        [attestation_service.attestation_token_broker]
        type = "Ear"
        policy_dir = "/opt/confidential-containers/attestation-service/policies"
    
        [attestation_service.attestation_token_broker.signer]
        key_path = "/etc/attestation-key/token.key"
        cert_path = "/etc/attestation-cert/token.crt"
    
        [attestation_service.attestation_token_config]
        duration_min = 5
    
        [attestation_service.rvps_config]
        type = "BuiltIn"
    
        [attestation_service.rvps_config.storage]
        type = "LocalJson"
        file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"
    
        [[plugins]]
        name = "resource"
        type = "LocalFs"
        dir_path = "/opt/confidential-containers/kbs/repository"
    
        [policy_engine]
        policy_path = "/opt/confidential-containers/opa/policy.rego"
    Copy to Clipboard Toggle word wrap
  2. Create the config map by running the following command:

    $ oc create -f kbs-config-cm.yaml
    Copy to Clipboard Toggle word wrap

3.7. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

  1. Create an rvps-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rvps-reference-values
      namespace: trustee-operator-system
    data:
      reference-values.json: |
        [
        ]
    Copy to Clipboard Toggle word wrap
  2. Create the RVPS config map by running the following command:

    $ oc create -f rvps-configmap.yaml
    Copy to Clipboard Toggle word wrap

3.8. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Open Policy Agent specification.

This policy checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 values against the reference values to ensure that the confidential containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. For details, see Linux TPM PCR Registry in the UAPI Group Specifications documentation.

Procedure

  1. Create an attestation-policy.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: attestation-policy
      namespace: trustee-operator-system
    data:
      default.rego: |
        package policy
    
        import rego.v1
        default executables := 33
        default hardware := 97
        default configuration := 36
    
        ##### Azure vTPM SNP
        executables := 3 if {
          input.azsnpvtpm.measurement in data.reference.measurement
          input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
        }
    
        hardware := 2 if {
          input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
          input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
          input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
          input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
        }
    
        configuration := 2 if {
          input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
          input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
          input.azsnpvtpm.policy_abi_major in data.reference.abi_major
          input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
          input.azsnpvtpm.policy_single_socket in data.reference.single_socket
          input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
        }
    
        ##### Azure vTPM TDX
        executables := 3 if {
          input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
          input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
          input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
          input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
          input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
        }
    
        hardware := 2 if {
          # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
          input.aztdxvtpm.quote.header.tee_type == "81000000"
          input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"
    
          # Check TDX Module version and its hash. Also check OVMF code hash.
          # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
          # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
          # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
        }
    
        configuration := 2 if {
          # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
        }
    Copy to Clipboard Toggle word wrap
  2. Create the attestation policy config map by running the following command:

    $ oc create -f attestation-policy.yaml
    Copy to Clipboard Toggle word wrap

3.9. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

  1. Create a tdx-config.yaml manifest file according to the following example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: tdx-config
      namespace: trustee-operator-system
    data:
      sgx_default_qcnl.conf: |
        {
          "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
        }
    Copy to Clipboard Toggle word wrap
  2. Create the tdx-config config map by running the following command:

    $ oc create -f tdx-config.yaml
    Copy to Clipboard Toggle word wrap

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

  • You have created one or more custom keys.

Procedure

  • Create a secret for the custom keys according to the following example:

    $ oc create secret generic attestation-status \
      --from-literal key1=<custom_key1> \ 
    1
    
      --from-literal key2=<custom_key2> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify a custom key.

    You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

  1. Create a secret for container image signature verification by running the following command:

    $ oc create secret generic <type> \ 
    1
    
      --from-file=<tag>=./<public_key_file> \ 
    2
    
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify the KBS secret type, for example, img-sig.
    2
    Specify the secret tag, for example, pub-key, and the public container image signing key.
  2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see containers-policy.json 5.

Note

Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.

Procedure

  1. Create a security-policy-config.json file according to the following example:

    {
      "default": [
          {
          "type": "reject" 
    1
    
          }
      ],
      "transports": {
          "<transport>": { 
    2
    
              "<registry>/<image>": 
    3
    
              [
                  {
                      "type": "sigstoreSigned",
                      "keyPath": "kbs:///default/<type>/<tag>" 
    4
    
                  }
              ]
          }
      }
    }
    Copy to Clipboard Toggle word wrap
    1
    By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
    2
    Specify the image repository for transport, for example, "docker":. For more information, see containers-transports 5.
    3
    Specify the container registry and image, for example, "quay.io/my-image".
    4
    Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
  2. Create the security policy by running the following command:

    $ oc create secret generic <security-policy-name> \
      --from-file=<osc-key>=./<security-policy-config.json> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

3.13. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

  1. Create a resourcepolicy-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: resource-policy
      namespace: trustee-operator-system
    data:
      policy.rego: |
        package policy
        default allow = true
        allow {
          input["submods"]["cpu0"]["ear.status"] == "affirming"
        }
    Copy to Clipboard Toggle word wrap
    policy.rego
    The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
    package policy
    The resource policy follows the Open Policy Agent specification.
  2. Create the resource policy config map by running the following command:

    $ oc create -f resourcepolicy-configmap.yaml
    Copy to Clipboard Toggle word wrap

3.14. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

  1. Create an edge route by running the following command:

    $ oc create route passthrough --service=kbs-service --port kbs-port \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Set the TRUSTEE_HOST variable by running the following command:

    $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
      -o jsonpath={.spec.host})
    Copy to Clipboard Toggle word wrap
  3. Verify the route by running the following command:

    $ echo $TRUSTEE_HOST
    Copy to Clipboard Toggle word wrap

    Example output

    kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
    Copy to Clipboard Toggle word wrap

3.15. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

  1. Create a private key by running the following command:

    $ openssl genpkey -algorithm ed25519 > privateKey
    Copy to Clipboard Toggle word wrap
  2. Create a public key by running the following command:

    $ openssl pkey -in privateKey -pubout -out publicKey
    Copy to Clipboard Toggle word wrap
  3. Create a secret by running the following command:

    $ oc create secret generic kbs-auth-public-key \
      --from-file=publicKey -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Verify the secret by running the following command:

    $ oc get secret -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

3.16. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

  1. Create a kbsconfig-cr.yaml manifest file:

    apiVersion: confidentialcontainers.org/v1alpha1
    kind: KbsConfig
    metadata:
      labels:
        app.kubernetes.io/name: kbsconfig
        app.kubernetes.io/instance: kbsconfig
        app.kubernetes.io/part-of: trustee-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: trustee-operator
      name: kbsconfig
      namespace: trustee-operator-system
    spec:
      kbsConfigMapName: kbs-config-cm
      kbsAuthSecretName: kbs-auth-public-key
      kbsDeploymentType: AllInOneDeployment
      kbsRvpsRefValuesConfigMapName: rvps-reference-values
      kbsSecretResources:
      - attestation-status
      - <security-policy-name>
      kbsResourcePolicyConfigMapName: resource-policy
      kbsHttpsKeySecretName: kbs-https-key
      kbsHttpsCertSecretName: kbs-https-certificate
      kbsAttestationCertSecretName: attestation-cert
      kbsAttestationKeySecretName: attestation-key
      kbsServiceType: NodePort
    # tdxConfigSpec:
    #   kbsTdxConfigMapName: tdx-config
    # kbsServiceType: <service_type>
    Copy to Clipboard Toggle word wrap
    • kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
    • Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
    • Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.
  2. Create the KbsConfig CR by running the following command:

    $ oc create -f kbsconfig-cr.yaml
    Copy to Clipboard Toggle word wrap

3.17. Updating the RVPS config map

You update the Reference Value Provider Service (RVPS) config map with expected measurements, including the Platform Configuration Register (PCR) 8 value, for the trusted execution environment. Red Hat build of Trustee uses these measurements to verify the attestation evidence.

The workload cluster administrator calculates the PCR8 value by performing a SHA256 hash on a configuration or policy file such as initdata:

$ hash=$(sha256sum <config_file> | cut -d' ' -f1)

$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1)
Copy to Clipboard Toggle word wrap

Prerequisites

  • PCR8 value, expiration, and algorithm, created by the workload cluster administrator

Procedure

  1. Create an rvps-configmap-update.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rvps-reference-values
      namespace: trustee-operator-system
    data:
      reference-values.json: |
        [
         {
            "name": "svn",
            "expiration": "2027-01-01T00:00:00Z",
            "value" : 1
          },
          {
            "name": "major_version",
            "expiration": "2027-01-01T00:00:00Z",
            "value" : 1
          },
          {
            "name": "minimum_minor_version",
            "expiration": "2027-01-01T00:00:00Z",
            "value" : 4
          }
        ]
    Copy to Clipboard Toggle word wrap
    Note

    Do not use this configuration example in a production environment.

    The "value" can be any JSON type (string, number, boolean, array, object). The JSON type must be combined with the operand used in the attestation-policy. See the following examples for valid "value" types:

    • If the attestation rule is:

      input.sample.platform_version.major == data.reference.major_version
      Copy to Clipboard Toggle word wrap

      The == operand expects to match an integer type in the reference values:

      {
      "name": "major_version",
      "expiration": "2027-01-01T00:00:00Z",
      "value" : 1
      }
      Copy to Clipboard Toggle word wrap
    • If the attestation rule is:

      input.sample.svn in data.reference.svn
      Copy to Clipboard Toggle word wrap

      The in operand expects to match an array type in the reference values:

      {
      "name": "svn",
      "expiration": "2027-01-01T00:00:00Z",
      "value" : [
      1
      ]
      }
      Copy to Clipboard Toggle word wrap
  2. Update the RVPS config map by running the following command:

    $ oc apply -f rvps-configmap-update.yaml
    Copy to Clipboard Toggle word wrap

3.18. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

  1. Set the default project by running the following command:

    $ oc project trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Check the pods by running the following command:

    $ oc get pods -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                   READY   STATUS    RESTARTS   AGE
    trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
    trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m
    Copy to Clipboard Toggle word wrap

  3. Set the POD_NAME environmental variable by running the following command:

    $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
    Copy to Clipboard Toggle word wrap
  4. Check the pod logs by running the following command:

    $ oc logs -n trustee-operator-system $POD_NAME
    Copy to Clipboard Toggle word wrap

    Example output

    [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
    [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
    [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
    [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
    [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
    [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime
    Copy to Clipboard Toggle word wrap

You can deploy Red Hat build of Trustee for confidential containers workloads running on IBM Z® and IBM® LinuxONE.

4.1. Prerequisites

4.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

  1. Install the Red Hat build of Trustee Operator.
  2. Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
  3. Create the Reference Value Provider Service (RVPS) config map.
  4. Configure the IBM Secure Execution certificates and keys.
  5. Create the IBM persistent storage components.
  6. Create the attestation policy config map.
  7. Optional: Create a secret for custom keys clients.
  8. Optional: Create a secret for container image signature verification.
  9. Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
  10. Create the resource policy config map.
  11. Create the KBSConfig CR.
  12. Create the cluster route.
  13. Create the authentication secret.
  14. Verify the Red Hat build of Trustee configuration.

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI tool (oc).

Procedure

  1. Create a trustee-namespace.yaml manifest file:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Create the trustee-operator-system namespace by running the following command:

    $ oc create -f trustee-namespace.yaml
    Copy to Clipboard Toggle word wrap
  3. Create a trustee-operatorgroup.yaml manifest file:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: trustee-operator-group
      namespace: trustee-operator-system
    spec:
      targetNamespaces:
      - trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Create the operator group by running the following command:

    $ oc create -f trustee-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap
  5. Create a trustee-subscription.yaml manifest file:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: trustee-operator-system
      namespace: trustee-operator-system
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: trustee-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    Copy to Clipboard Toggle word wrap
  6. Create the subscription by running the following command:

    $ oc create -f trustee-subscription.yaml
    Copy to Clipboard Toggle word wrap
  7. Verify that the Operator is correctly installed by running the following command:

    $ oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command:

    $ watch oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                      DISPLAY                        PHASE
    trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded
    Copy to Clipboard Toggle word wrap

4.4. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Note

The following configuration example turns off security features. Do not use this example in a production environment.

Procedure

  1. Create a kbs-config-cm.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: kbs-config-cm
      namespace: trustee-operator-system
    data:
      kbs-config.toml: |
        [http_server]
        sockets = ["0.0.0.0:8080"]
        insecure_http = false
        private_key = "/etc/https-key/tls.key"
        certificate = "/etc/https-cert/tls.crt"
        worker_count = 4
    
        [admin]
        insecure_api = false
    
        auth_public_key = "/etc/auth-secret/publicKey"
    
        [attestation_token]
        insecure_key = true
        attestation_token_type = "CoCo"
    
        [attestation_service]
        type = "coco_as_builtin"
        work_dir = "/opt/confidential-containers/attestation-service"
        policy_engine = "opa"
    
        [attestation_service.attestation_token_broker]
        type = "Simple"
        policy_dir = "/opt/confidential-containers/attestation-service/policies"
    
    
        [attestation_service.attestation_token_config]
        duration_min = 5
    
        [attestation_service.rvps_config]
        type = "BuiltIn"
    
        [attestation_service.rvps_config.storage]
        type = "LocalJson"
        file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"
    
        [[plugins]]
        name = "resource"
        type = "LocalFs"
        dir_path = "/opt/confidential-containers/kbs/repository"
    
        [policy_engine]
        policy_path = "/opt/confidential-containers/opa/policy.rego"
    Copy to Clipboard Toggle word wrap
  2. Create the config map by running the following command:

    $ oc create -f kbs-config-cm.yaml
    Copy to Clipboard Toggle word wrap

4.5. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

  1. Create an rvps-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rvps-reference-values
      namespace: trustee-operator-system
    data:
      reference-values.json: |
        [
        ]
    Copy to Clipboard Toggle word wrap
  2. Create the RVPS config map by running the following command:

    $ oc create -f rvps-configmap.yaml
    Copy to Clipboard Toggle word wrap

You must configure the IBM Secure Execution (SE) certificates and keys for your worker nodes.

Prerequisites

  • You have the IP address of the bastion node.
  • You have the internal IP addresses of the worker nodes.

Procedure

  1. Generate the Key Broker Service (KBS) certificate and key.
  2. Obtain the attestation policy fields.
  3. Download the certificates and certificate revocation lists (CRLs).
  4. Generate the RSA keys.
  5. Verify and copy files to the OpenShift Container Platform worker nodes.
  6. Create the secrets in the cluster with the KBS key and certificate.

You must generate the Key Broker Service (KBS) certificate and key.

Procedure

  1. Create the kbs.conf configuration file according to the following example:

    [req]
    default_bits = 2048
    default_keyfile = localhost.key
    distinguished_name = req_distinguished_name
    req_extensions = req_ext
    x509_extensions = v3_ca
    
    [req_distinguished_name]
    countryName = Country Name (2-letter code)
    countryName_default = <country_name>
    stateOrProvinceName = State or Province Name (full name)
    stateOrProvinceName_default = <state_name>
    localityName = Locality Name (eg, city)
    localityName_default = <locality_name>
    organizationName = Organization Name (eg, company)
    organizationName_default = Red Hat
    organizationalUnitName = organizationalunit
    organizationalUnitName_default = Development
    commonName = Common Name (e.g. server FQDN or
    YOUR name)
    commonName_default = kbs-service
    commonName_max = 64
    
    [req_ext]
    subjectAltName = @alt_names
    
    [v3_ca]
    subjectAltName = @alt_names
    
    [alt_names]
    IP.1  = <worker_node_ip>
    DNS.1  = localhost
    DNS.2  = 127.0.0.1
    Copy to Clipboard Toggle word wrap
    <worker_node_ip>

    Obtain the IP address of a worker node by running the following command:

    $ oc get node $(oc get pod -n trustee-operator-system -o jsonpath='{.items[0].spec.nodeName}') -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'
    Copy to Clipboard Toggle word wrap
  2. Generate the KBS key and self-signed certificate by running the following command:

    $ openssl req -x509 -nodes -days 365 \
       -newkey rsa:2048 \
       -keyout kbs.key \
       -out kbs.crt \
       -config kbs.conf \
       -passin pass:
    Copy to Clipboard Toggle word wrap
  3. Copy the KBS key to the ibmse directory by running the following command:

    $ cp kbs.key /tmp/ibmse/kbs.key
    Copy to Clipboard Toggle word wrap
  4. Copy the KBS certificate to the ibmse directory by running the following command:

    $ cp kbs.crt /tmp/ibmse/kbs.crt
    Copy to Clipboard Toggle word wrap

4.6.2. Obtaining the attestation policy fields

You must obtain the attestation policy fields by using Reference Value Provider Service (RVPS).

Procedure

  1. Create a directory to download the GetRvps.sh script by running the following command:

    $ mkdir -p Rvps-Extraction/
    Copy to Clipboard Toggle word wrap
  2. Download the script by running the following command:

    $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/GetRvps.sh -O $PWD/GetRvps.sh
    Copy to Clipboard Toggle word wrap
  3. Create a subdirectory by running the following command:

    $ mkdir -p Rvps-Extraction/static-files
    Copy to Clipboard Toggle word wrap
  4. Go to the static-files directory by running the following command:

    $ cd Rvps-Extraction/static-files
    Copy to Clipboard Toggle word wrap
  5. Download the pvextract-hdr tool by running the following command:

    $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/pvextract-hdr -O $PWD/pvextract-hdr
    Copy to Clipboard Toggle word wrap
  6. Make the tool executable by running the following command:

    $ chmod +x pvextract-hdr
    Copy to Clipboard Toggle word wrap
  7. Download the se_parse_hdr.py script by running the following command:

    $ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/se_parse_hdr.py -O $PWD/se_parse_hdr.py
    Copy to Clipboard Toggle word wrap
  8. Copy your Host Key Document (HKD) certificate to the static-files directory by running the following command:

    $ cp ~/path/to/<hkd_cert.crt> .
    Copy to Clipboard Toggle word wrap

    The static-files directory contains the following files:

    • HKD.crt
    • pvextract-hdr
    • se_parse_hdr.py
  9. Go to the Rvps-Extraction directory by running the following command:

    $ cd ..
    Copy to Clipboard Toggle word wrap
  10. Make the GetRvps.sh script executable by running the following command:

    $ chmod +x GetRvps.sh
    Copy to Clipboard Toggle word wrap
  11. Run the script:

    $ ./GetRvps.sh
    Copy to Clipboard Toggle word wrap

    Example output

    ***Installing necessary packages for RVPS values extraction ***
    Updating Subscription Management repositories.
    Last metadata expiration check: 0:37:12 ago on Mon Nov 18 09:20:29 2024.
    Package python3-3.9.19-8.el9_5.1.s390x is already installed.
    Package python3-cryptography-36.0.1-4.el9.s390x is already installed.
    Package kmod-28-10.el9.s390x is already installed.
    Dependencies resolved.
    Nothing to do.
    Complete!
    ***Installation Finished ***
    1) Generate the RVPS From Local Image from User pc
    2) Generate RVPS from Volume
    3) Quit
    Please enter your choice:
    Copy to Clipboard Toggle word wrap

    1. Enter 2 to generate the Reference Value Provider Service from the volume:

      Please enter your choice: 2
      Copy to Clipboard Toggle word wrap
    2. Enter fa-pp for the libvirt pool name:

      Enter the Libvirt Pool Name: fa-pp
      Copy to Clipboard Toggle word wrap
    3. Enter the libvirt gateway URI:

      Enter the Libvirt URI Name: <libvirt-uri> 
      1
      Copy to Clipboard Toggle word wrap
      1
      Specify the LIBVIRT_URI value that you used to create the peer pods secret.
    4. Enter fa-pp-vol for the libvirt volume name:

      Enter the Libvirt Volume Name: fa-pp-vol
      Copy to Clipboard Toggle word wrap

      Example output

      Downloading from PODVM Volume...
      
      mount: /mnt/myvm: special device /dev/nbd3p1 does not exist.
      Error: Failed to mount the image. Retrying...
      Mounting on second attempt passed
      /dev/nbd3 disconnected
      SE header found at offset 0x014000
      SE header written to '/root/Rvps-Extraction/output-files/hdr.bin' (640 bytes)
      se.tag:  42f3fe61e8a7e859cab3bb033fd11c61
      se.image_phkh:  92d0aff6eb86719b6b1ea0cb98d2c99ff2ec693df3efff2158f54112f6961508
      provenance = ewogICAgInNlLmF0dGVzdGF0aW9uX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnRhZyI6IFsKICAgICAgICAiNDJmM2ZlNjFlOGE3ZTg1OWNhYjNiYjAzM2ZkMTFjNjEiCiAgICBdLAogICAgInNlLmltYWdlX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnVzZXJfZGF0YSI6IFsKICAgICAgICAiMDAiCiAgICBdLAogICAgInNlLnZlcnNpb24iOiBbCiAgICAgICAgIjI1NiIKICAgIF0KfQo=
      -rw-r--r--. 1 root root 640 Dec 16 10:57 /root/Rvps-Extraction/output-files/hdr.bin
      -rw-r--r--. 1 root root 446 Dec 16 10:57 /root/Rvps-Extraction/output-files/ibmse-policy.rego
      -rw-r--r--. 1 root root 561 Dec 16 10:57 /root/Rvps-Extraction/output-files/se-message
      Copy to Clipboard Toggle word wrap

    5. Obtain the attestation policy field values by running the following command:

      $ cat /root/Rvps-Extraction/output-files/se-sample
      Copy to Clipboard Toggle word wrap

      Example output

      {
          "se.attestation_phkh": [
              "50a59219c5034f23f69a81893b77f80190dab0ab4781d10b6631d6ed23ef38e4"
          ],
          "se.tag": [
              "ded333bce2d721547ee2b59b1b96e7e5"
          ],
          "se.image_phkh": [
              "50a59219c5034f23f69a81893b77f80190dab0ab4781d10b6631d6ed23ef38e4"
          ],
          "se.user_data": [
              "00"
          ],
          "se.version": [
              "256"
          ]
      }
      Copy to Clipboard Toggle word wrap

You must download the IBM certificates and Certificate Revocation Lists (CRLs).

Procedure

  1. Create a temporary directory for certificates by running the following command:

    $ mkdir /tmp/ibmse/certs
    Copy to Clipboard Toggle word wrap
  2. Download the ibm-z-host-key-signing-gen2.crt certificate by running the following command:

    $ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt
    Copy to Clipboard Toggle word wrap
  3. Download the DigiCertCA.crt certificate by running the following command:

    $ wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt
    Copy to Clipboard Toggle word wrap
  4. Create a temporary directory for the CRLs by running the following command:

    $ mkdir /tmp/ibmse/crls
    Copy to Clipboard Toggle word wrap
  5. Download the ibm-z-host-key-gen2.crl file by running the following command:

    $ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-gen2.crl -O /tmp/ibmse/crls/ibm-z-host-key-gen2.crl
    Copy to Clipboard Toggle word wrap
  6. Download the DigiCertTrustedRootG4.crl file by running the following command:

    $ wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl
    Copy to Clipboard Toggle word wrap
  7. Download the DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl file by running the following command:

    $ wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
    Copy to Clipboard Toggle word wrap
  8. Create a temporary directory for the hdr.bin file by running the following command:

    $ mkdir -p /tmp/ibmse/hdr/
    Copy to Clipboard Toggle word wrap
  9. Copy the hdr.bin file to the hdr directory by running the following command:

    $ cp /root/Rvps-Extraction/output-files/hdr.bin /tmp/ibmse/hdr/
    Copy to Clipboard Toggle word wrap
  10. Create a temporary directory for Host Key Document (HKD) certificate by running the following command:

    $ mkdir -p /tmp/ibmse/hkds
    Copy to Clipboard Toggle word wrap
  11. Copy your HKD certificate to the hkds directory by running the following command:

    $ cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/
    Copy to Clipboard Toggle word wrap

4.6.4. Generating the RSA keys

You must generate the RSA key pair for encryption.

Procedure

  1. Generate an RSA key pair by running the following command:

    $ openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 
    1
    Copy to Clipboard Toggle word wrap
  2. Create a temporary directory for the RSA keys by running the following command:

    $ mkdir -p /tmp/ibmse/rsa
    Copy to Clipboard Toggle word wrap
  3. Create an encrypt_key.pub key by running the following command:

    $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub
    Copy to Clipboard Toggle word wrap
  4. Create an encrypt_key.pem key by running the following command:

    $ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem
    Copy to Clipboard Toggle word wrap

You must verify the /tmp/ibmse folder structure and copy the files to the OpenShift Container Platform worker nodes.

Procedure

  1. Verify the structure of the /tmp/ibmse directory by running the following command:

    $ tree /tmp/ibmse
    Copy to Clipboard Toggle word wrap

    Example output

    /tmp/ibmse
    ├──kbs.key
    ├──kbs.crt
    ├── certs
      ├── ibm-z-host-key-signing-gen2.crt
      └── DigiCertCA.crt
    ├── crls
      └── ibm-z-host-key-gen2.crl
      └── DigiCertTrustedRootG4.crl
      └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
    ├── hdr
      └── hdr.bin
    ├── hkds
      └── <hkd_cert.crt>
    ├── rsa
      └── encrypt_key.pem
      └── encrypt_key.pub
    Copy to Clipboard Toggle word wrap

  2. Create a compressed file from the /tmp/ibmse directory by running the following command:

    $ tar -czf ibmse.tar.gz -C /tmp/ibmse
    Copy to Clipboard Toggle word wrap
  3. Copy the .tar.gz file to the bastion node in your cluster by running the following command:

    $ scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 
    1
    Copy to Clipboard Toggle word wrap
  4. Connect to the bastion node over SSH by running the following command:

    $ ssh root@<ocp_bastion_ip>
    Copy to Clipboard Toggle word wrap
  5. Copy the .tar.gz file to each worker node by running the following command:

    $ scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 
    1
    Copy to Clipboard Toggle word wrap
  6. Extract the .tar.gz on each worker node by running the following command:

    $ ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/'
    Copy to Clipboard Toggle word wrap
  7. Update the ibmse folder permissions by running the following command:

    $ ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/'
    Copy to Clipboard Toggle word wrap

4.6.6. Creating the secrets for Key Broker Service

You must create the secrets in the cluster with the Key Broker Service (KBS) key and certificate.

Procedure

  1. Convert the kbs.crt file to a Base64-encoded string by running the following command:

    $ cat /tmp/ibmse/kbs.crt | base64 -w 0
    Copy to Clipboard Toggle word wrap
  2. Record the string for the kbs-https-certificate.yaml manifest.
  3. Create a kbs-https-certificate.yaml manifest file according to the following example:

    apiVersion: v1
    kind: Secret
    metadata:
     name: kbs-https-certificate
     namespace: trustee-operator-system
    data:
     https.crt: <kbs_crt_value>
    Copy to Clipboard Toggle word wrap
    <kbs_crt_value>
    Specify the Base64-encoded string you created from the kbs.crt file.
  4. Create the secret with the KBS certificate by running the following command:

    $ oc create -f kbs-https-certificate.yaml
    Copy to Clipboard Toggle word wrap
  5. Convert the kbs.key file to a Base64-encoded string by running the following command:

    $ cat /tmp/ibmse/kbs.key | base64 -w 0
    Copy to Clipboard Toggle word wrap
  6. Record the string for the kbs-https-key.yaml manifest.
  7. Create a kbs-https-key.yaml manifest file according to the following example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: kbs-https-key
      namespace: trustee-operator-system
    data:
     https.key: <kbs_key_value>
    Copy to Clipboard Toggle word wrap
    <kbs_key_value>
    Specify the Base64-encoded string you created from the kbs.key file.
  8. Create the secret with the KBS key by running the following command:

    $ oc create -f kbs-https-key.yaml
    Copy to Clipboard Toggle word wrap

You must create the persistent volume (PV) and persistent volume claim (PVC) to mount the ibmse folder on the trustee-deployment pod.

Procedure

  1. Create a persistent-volume.yaml manifest file:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: ibmse-pv
      namespace: trustee-operator-system
    spec:
      capacity:
        storage: 100Mi
      accessModes:
        - ReadOnlyMany
      storageClassName: ""
      local:
        path: /opt/confidential-containers/ibmse
      nodeAffinity:
        required:
          nodeSelectorTerms:
            - matchExpressions:
                - key: node-role.kubernetes.io/worker
                  operator: Exists
    Copy to Clipboard Toggle word wrap
  2. Create the persistent volume by running the following command:

    $ oc create -f persistent-volume.yaml
    Copy to Clipboard Toggle word wrap
  3. Create a persistent-volume-claim.yaml manifest file:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: ibmse-pvc
      namespace: trustee-operator-system
    spec:
      accessModes:
        - ReadOnlyMany
      storageClassName: ""
      resources:
        requests:
          storage: 100Mi
    Copy to Clipboard Toggle word wrap
  4. Create the persistent volume claim by running the following command:

    $ oc create -f persistent-volume-claim.yaml
    Copy to Clipboard Toggle word wrap

4.8. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Open Policy Agent specification.

Procedure

  1. Create an attestation-policy.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: attestation-policy
      namespace: trustee-operator-system
    data:
      default.rego: |
        package policy
        import rego.v1
        default allow = false
        converted_version := sprintf("%v", [input["se.version"]])
        allow if {
            input["se.attestation_phkh"] == "<se.attestation_phkh>"
            input["se.image_phkh"] == "<se.image_phkh>"
            input["se.tag"] == "<se.tag>"
            converted_version == "256"
        }
    Copy to Clipboard Toggle word wrap
    default.rego
    Do not modify the policy name.
  2. Create the attestation policy config map by running the following command:

    $ oc create -f attestation-policy.yaml
    Copy to Clipboard Toggle word wrap

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

  • You have created one or more custom keys.

Procedure

  • Create a secret for the custom keys according to the following example:

    $ oc create secret generic attestation-status \
      --from-literal key1=<custom_key1> \ 
    1
    
      --from-literal key2=<custom_key2> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1 1 1 1
    Specify a custom key.

    You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

  1. Create a secret for container image signature verification by running the following command:

    $ oc create secret generic <type> \ 
    1
    
      --from-file=<tag>=./<public_key_file> \ 
    2
    
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    1
    Specify the KBS secret type, for example, img-sig.
    2
    Specify the secret tag, for example, pub-key, and the public container image signing key.
  2. Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see containers-policy.json 5.

Note

Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.

Procedure

  1. Create a security-policy-config.json file according to the following example:

    {
        "default": [
        ],
        "transports": {
            "docker": {
                "<container_registry_url>/<username>/busybox:latest": 
    1
    
                [
                    {
                        "type": "sigstoreSigned",
                        "keyPath": "kbs:///default/img-sig/pub-key" 
    2
    
                    }
                ]
            }
        }
    }
    Copy to Clipboard Toggle word wrap
    1
    Specify the container registry URL, for example, "quay.io".
    2
    Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
  2. Create the security policy by running the following command:

    $ oc create secret generic <security-policy-name> \
      --from-file=<osc-key>=./<security-policy-config.json> \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

4.12. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

  1. Create a resourcepolicy-configmap.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: resource-policy
      namespace: trustee-operator-system
    data:
      policy.rego: |
        package policy
        default allow = true
        allow {
          input["tee"] == "se"
        }
    Copy to Clipboard Toggle word wrap
    policy.rego
    The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
    package policy
    The resource policy follows the Open Policy Agent specification.
  2. Create the resource policy config map by running the following command:

    $ oc create -f resourcepolicy-configmap.yaml
    Copy to Clipboard Toggle word wrap

4.13. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

  1. Create an edge route by running the following command:

    $ oc create route passthrough --service=kbs-service --port kbs-port \
      -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Set the TRUSTEE_HOST variable by running the following command:

    $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
      -o jsonpath={.spec.host})
    Copy to Clipboard Toggle word wrap
  3. Verify the route by running the following command:

    $ echo $TRUSTEE_HOST
    Copy to Clipboard Toggle word wrap

    Example output

    kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
    Copy to Clipboard Toggle word wrap

4.14. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

  1. Create a private key by running the following command:

    $ openssl genpkey -algorithm ed25519 > privateKey
    Copy to Clipboard Toggle word wrap
  2. Create a public key by running the following command:

    $ openssl pkey -in privateKey -pubout -out publicKey
    Copy to Clipboard Toggle word wrap
  3. Create a secret by running the following command:

    $ oc create secret generic kbs-auth-public-key \
      --from-file=publicKey -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  4. Verify the secret by running the following command:

    $ oc get secret -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

4.15. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

  1. Create a kbsconfig-cr.yaml manifest file:

    apiVersion: confidentialcontainers.org/v1alpha1
    kind: KbsConfig
    metadata:
      labels:
        app.kubernetes.io/name: kbsconfig
        app.kubernetes.io/instance: kbsconfig
        app.kubernetes.io/part-of: trustee-operator
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/created-by: trustee-operator
      name: kbsconfig
      namespace: trustee-operator-system
    spec:
      kbsConfigMapName: kbs-config-cm
      kbsAuthSecretName: kbs-auth-public-key
      kbsDeploymentType: AllInOneDeployment
      kbsRvpsRefValuesConfigMapName: rvps-reference-values
      kbsSecretResources:
      - attestation-status
      - <security-policy-name>
      kbsResourcePolicyConfigMapName: resource-policy
      kbsHttpsKeySecretName: kbs-https-key
      kbsHttpsCertSecretName: kbs-https-certificate
      kbsAttestationCertSecretName: attestation-cert
      kbsAttestationKeySecretName: attestation-key
      kbsServiceType: NodePort
      ibmSEConfigSpec:
        certStorePvc: ibmse-pvc
      KbsEnvVars:
        SE_SKIP_CERTS_VERIFICATION: "false"
    Copy to Clipboard Toggle word wrap
    • kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
    • SE_SKIP_CERTS_VERIFICATION: Set to true only for testing purposes.
  2. Create the KbsConfig CR by running the following command:

    $ oc create -f kbsconfig-cr.yaml
    Copy to Clipboard Toggle word wrap

4.16. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

  1. Set the default project by running the following command:

    $ oc project trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Check the pods by running the following command:

    $ oc get pods -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                   READY   STATUS    RESTARTS   AGE
    trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
    trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m
    Copy to Clipboard Toggle word wrap

  3. Set the POD_NAME environmental variable by running the following command:

    $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
    Copy to Clipboard Toggle word wrap
  4. Check the pod logs by running the following command:

    $ oc logs -n trustee-operator-system $POD_NAME
    Copy to Clipboard Toggle word wrap

    Example output

    [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
    [2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
    [2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
    [2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
    [2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
    [2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime
    Copy to Clipboard Toggle word wrap

Chapter 5. Uninstalling

You uninstall by performing the following tasks:

  1. Delete the KbsConfig custom resource.
  2. Uninstall the Red Hat build of Trustee Operator.
  3. Delete the KbsConfig custom resource definition.

5.1. Deleting the KbsConfig custom resource

You delete the KbsConfig custom resource (CR) by using the command line.

Procedure

  1. Delete the KbsConfig CR by running the following command:

    $ oc delete kbsconfig kbsconfig
    Copy to Clipboard Toggle word wrap
  2. Verify the CR removal by running the following command:

    $ oc get kbsconfig kbsconfig
    Copy to Clipboard Toggle word wrap

    Example output

    No kbsconfig instances exist
    Copy to Clipboard Toggle word wrap

Important

You must ensure that all pods are deleted. Any remaining pod resources might result in an unexpected bill from your cloud provider.

You uninstall the Red Hat build of Trustee Operator by using the command line.

Procedure

  1. Delete the subscription by running the following command:

    $ oc delete subscription trustee-operator -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
  2. Delete the namespace by running the following command:

    $ oc delete namespace trustee-operator-system
    Copy to Clipboard Toggle word wrap

5.3. Deleting the KbsConfig CRD

You delete the KbsConfig custom resource definition (CRD) by using the command line.

Prerequisites

  • You have deleted the KbsConfig custom resource.
  • You have uninstalled the Red Hat build of Trustee Operator.

Procedure

  1. Delete the KbsConfig CRD by running the following command:

    $ oc delete crd kataconfigs.kataconfiguration.openshift.io
    Copy to Clipboard Toggle word wrap
  2. Verify that the CRD was deleted by running the following command:

    $ oc get crd kataconfigs.kataconfiguration.openshift.io
    Copy to Clipboard Toggle word wrap

    Example output

    Unknown CRD kataconfigs.kataconfiguration.openshift.io
    Copy to Clipboard Toggle word wrap

Legal Notice

Copyright © Red Hat.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red Hat