Chapter 3. Deploying Red Hat build of Trustee for workloads running on Microsoft Azure Red Hat OpenShift
You can deploy Red Hat build of Trustee for confidential containers workloads running on Azure Red Hat OpenShift.
3.1. Prerequisites Copy linkLink copied to clipboard!
- You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing OpenShift Container Platform on bare metal.
3.2. Deployment overview Copy linkLink copied to clipboard!
You deploy Red Hat build of Trustee by performing the following steps:
- Install the Red Hat build of Trustee Operator.
- Create HTTPS secrets.
- Create the attestation token secret.
-
Optional: Create the
kbs-configconfig map if you are using Intel® Trust Domain Extensions (TDX) remote attestation. -
Create the Reference Value Provider Service (RVPS) config map. Initially, you create an empty config map for the reference values. You update the values after you create
KBSConfigcustom resource (CR). - Create the attestation policy config map.
- Optional: Create a config map for Intel® TDX.
- Optional: Create a secret for custom keys clients.
- Optional: Create a secret for container image signature verification.
- Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
- Create the resource policy config map.
-
Create the
KBSConfigCR. - Create the cluster route.
- Create the authentication secret.
- Update the RVPS config map with the reference values.
- Verify the Red Hat build of Trustee configuration.
3.3. Installing the Red Hat build of Trustee Operator Copy linkLink copied to clipboard!
You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.
Prerequisites
-
You have access to the cluster as a user with the
cluster-adminrole. -
You have installed the OpenShift CLI tool (
oc).
Procedure
Create a
trustee-namespace.yamlmanifest file:apiVersion: v1 kind: Namespace metadata: name: trustee-operator-systemCreate the
trustee-operator-systemnamespace by running the following command:$ oc create -f trustee-namespace.yamlCreate a
trustee-operatorgroup.yamlmanifest file:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: trustee-operator-group namespace: trustee-operator-system spec: targetNamespaces: - trustee-operator-systemCreate the operator group by running the following command:
$ oc create -f trustee-operatorgroup.yamlCreate a
trustee-subscription.yamlmanifest file:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: trustee-operator-system namespace: trustee-operator-system spec: channel: stable installPlanApproval: Automatic name: trustee-operator source: redhat-operators sourceNamespace: openshift-marketplaceCreate the subscription by running the following command:
$ oc create -f trustee-subscription.yamlVerify that the Operator is correctly installed by running the following command:
$ oc get csv -n trustee-operator-systemThis command can take several minutes to complete.
Watch the process by running the following command:
$ watch oc get csv -n trustee-operator-systemExample output
NAME DISPLAY PHASE trustee-operator.v1.0.0 Trustee Operator 1.0.0 Succeeded
3.4. Creating HTTPS secrets Copy linkLink copied to clipboard!
Generate keys to securely launch Red Hat build of Trustee and enables services to use HTTPS.
Procedure
Set the
DOMAINvariable for the cluster by running the following command:$ DOMAIN=$(oc get ingress.config/cluster -o jsonpath='{.spec.domain}')Set the
NSvariable for the Red Hat build of Trustee namespace by running the following command:$ NS=trustee-operator-systemSet the
ROUTE_NAMEvariable by running the following command:$ ROUTE_NAME=kbs-serviceSet the
ROUTEvariable to the full DNS name by running the following command:$ ROUTE="${ROUTE_NAME}-${NS}.${DOMAIN}"Generate a private SSL/TLS key and certificate for Red Hat build of Trustee by running the following command:
$ openssl req -x509 -nodes -days 365 \ -newkey rsa:2048 \ -keyout tls.key \ -out tls.crt \ -subj "/CN=<custom_cn>/O=<custom_org>" \ -addext "subjectAltName=DNS:${ROUTE}"-
<custom_cn>is a custom CN. For example:kbs-trustee-operator-system. -
<custom_org>is a name of your organization.
-
Create the
kbs-https-certificatesecret in thetrustee-operator-systemnamespace by running the following command:$ oc create secret generic kbs-https-certificate --from-file=tls.crt -n trustee-operator-systemCreate the
kbs-https-keysecret in thetrustee-operator-systemnamespace by running the following command:$ oc create secret generic kbs-https-key --from-file=tls.key -n trustee-operator-system
3.5. Creating the attestation token secrets Copy linkLink copied to clipboard!
Generate an attestation token key and certificate for Red Hat build of Trustee.
Procedure
Generate a private elliptic curve SSL key called
token.keyby running the following command:$ openssl ecparam -name prime256v1 -genkey -noout -out token.keyCreate the
attestation-keysecret from the SSL/TLS key in thetrustee-operator-systemnamespace:$ oc create secret generic attestation-key \ --from-file=token.key \ -n trustee-operator-systemGenerate a self-signed SSL/TLS certificate from the private SSL key by running the following command:
$ openssl req -new -x509 -key token.key -out token.crt -days 365 \ -subj "/CN=<custom_cn>/O=<custom_org>"-
<custom_cn>: Specify the Common Name. For example:kbs-trustee-operator-system. -
<custom_org>: Specify your organization name.
-
Create the
attestation-certsecret from the SSL/TLS key and certificate in thetrustee-operator-systemnamespace:$ oc create secret generic attestation-cert \ --from-file=token.crt \ -n trustee-operator-systemCreate the
attestation-statussecret used for verifying the attestation process:$ oc create secret generic attestation-status \ --from-literal=status=success \ -n trustee-operator-system
3.6. Creating the kbs-config config map Copy linkLink copied to clipboard!
You create the kbs-config config map to configure Red Hat build of Trustee.
Procedure
Create a
kbs-config-cm.yamlmanifest file:apiVersion: v1 kind: ConfigMap metadata: name: kbs-config-cm namespace: trustee-operator-system data: kbs-config.toml: | [http_server] sockets = ["0.0.0.0:8080"] insecure_http = false private_key = "/etc/https-key/tls.key" certificate = "/etc/https-cert/tls.crt" worker_count = 4 [admin] insecure_api = false auth_public_key = "/etc/auth-secret/publicKey" [attestation_token] insecure_key = false trusted_certs_paths = ["/etc/attestation-cert/token.crt"] attestation_token_type = "CoCo" [attestation_service] type = "coco_as_builtin" work_dir = "/opt/confidential-containers/attestation-service" policy_engine = "opa" [attestation_service.attestation_token_broker] type = "Ear" policy_dir = "/opt/confidential-containers/attestation-service/policies" [attestation_service.attestation_token_broker.signer] key_path = "/etc/attestation-key/token.key" cert_path = "/etc/attestation-cert/token.crt" [attestation_service.attestation_token_config] duration_min = 5 [attestation_service.rvps_config] type = "BuiltIn" [attestation_service.rvps_config.storage] type = "LocalJson" file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json" [[plugins]] name = "resource" type = "LocalFs" dir_path = "/opt/confidential-containers/kbs/repository" [policy_engine] policy_path = "/opt/confidential-containers/opa/policy.rego"Create the config map by running the following command:
$ oc create -f kbs-config-cm.yaml
3.7. Creating the RVPS config map Copy linkLink copied to clipboard!
You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).
The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.
The data.reference-values.json stanza must be present, but it can be empty.
Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.
Procedure
Create an
rvps-configmap.yamlmanifest file:apiVersion: v1 kind: ConfigMap metadata: name: rvps-reference-values namespace: trustee-operator-system data: reference-values.json: | [ ]Create the RVPS config map by running the following command:
$ oc create -f rvps-configmap.yaml
3.8. Creating the attestation policy config map Copy linkLink copied to clipboard!
You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.
The attestation policy follows the Open Policy Agent specification.
This policy checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 values against the reference values to ensure that the confidential containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. For details, see Linux TPM PCR Registry in the UAPI Group Specifications documentation.
Procedure
Create an
attestation-policy.yamlmanifest file:apiVersion: v1 kind: ConfigMap metadata: name: attestation-policy namespace: trustee-operator-system data: default_cpu.rego: | package policy import rego.v1 default executables := 33 default hardware := 97 default configuration := 36 ##### Azure vTPM SNP executables := 3 if { input.azsnpvtpm.measurement in data.reference.measurement input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11 } hardware := 2 if { input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee } configuration := 2 if { input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled input.azsnpvtpm.policy_abi_major in data.reference.abi_major input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor input.azsnpvtpm.policy_single_socket in data.reference.single_socket input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed } ##### Azure vTPM TDX executables := 3 if { input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03 input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08 input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09 input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11 input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12 } hardware := 2 if { # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave input.aztdxvtpm.quote.header.tee_type == "81000000" input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607" # Check TDX Module version and its hash. Also check OVMF code hash. # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td } configuration := 2 if { # input.aztdxvtpm.quote.body.xfam in data.reference.xfam }Create the attestation policy config map by running the following command:
$ oc create -f attestation-policy.yaml
3.9. Creating a tdx-config config map Copy linkLink copied to clipboard!
Create a config map for Intel® Trust Domain Extensions (TDX).
Procedure
Create a
tdx-config.yamlmanifest file according to the following example:apiVersion: v1 kind: ConfigMap metadata: name: tdx-config namespace: trustee-operator-system data: sgx_default_qcnl.conf: | { "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/" }Create the
tdx-configconfig map by running the following command:$ oc create -f tdx-config.yaml
3.10. Creating a secret with custom keys for clients Copy linkLink copied to clipboard!
You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.
In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.
Prerequisites
- You have created one or more custom keys.
Procedure
Create a secret for the custom keys according to the following example:
$ oc create secret generic attestation-status \ --from-literal key1=<custom_key1> \1 --from-literal key2=<custom_key2> \ -n trustee-operator-system- 1
- Specify a custom key.
You specify the
attestation-statussecret in thespec.kbsSecretResourceskey of theKbsConfigcustom resource manifest.
3.11. Creating a secret for container image signature verification Copy linkLink copied to clipboard!
If you use container image signature verification, you must create a secret that contains the public container image signing key.
The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.
You can use Red Hat Trusted Artifact Signer or other tools to sign container images.
Procedure
Create a secret for container image signature verification by running the following command:
$ oc create secret generic <type> \1 --from-file=<tag>=./<public_key_file> \2 -n trustee-operator-system-
Record the
<type>value. You must add this value to thespec.kbsSecretResourceskey when you create theKbsConfigcustom resource.
3.12. Creating the container image signature verification policy Copy linkLink copied to clipboard!
You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see containers-policy.json 5.
Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.
Procedure
Create a
security-policy-config.jsonfile according to the following example:{ "default": [ { "type": "reject"1 } ], "transports": { "<transport>": {2 "<registry>/<image>":3 [ { "type": "sigstoreSigned", "keyPath": "kbs:///default/<type>/<tag>"4 } ] } } }- 1
- By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
- 2
- Specify the image repository for
transport, for example,"docker":. For more information, see containers-transports 5. - 3
- Specify the container registry and image, for example, "quay.io/my-image".
- 4
- Specify the type and tag of the container image signature verification secret that you created, for example,
img-sig/pub-key.
Create the security policy by running the following command:
$ oc create secret generic <security-policy-name> \ --from-file=<osc-key>=./<security-policy-config.json> \ -n trustee-operator-systemThe
<security-policy-name>secret is specified in thespec.kbsSecretResourceskey of theKbsConfigcustom resource.
3.13. Creating the resource policy config map Copy linkLink copied to clipboard!
You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.
This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.
Procedure
Create a
resourcepolicy-configmap.yamlmanifest file:apiVersion: v1 kind: ConfigMap metadata: name: resource-policy namespace: trustee-operator-system data: policy.rego: | package policy default allow = true allow { input["submods"]["cpu0"]["ear.status"] == "affirming" }- policy.rego
-
The name of the resource policy,
policy.rego, must match the resource policy defined in thekbs-configconfig map. - package policy
- The resource policy follows the Open Policy Agent specification.
Create the resource policy config map by running the following command:
$ oc create -f resourcepolicy-configmap.yaml
3.14. Creating the cluster route Copy linkLink copied to clipboard!
You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.
External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.
Procedure
Create an edge route by running the following command:
$ oc create route passthrough --service=kbs-service --port kbs-port \ -n trustee-operator-systemSet the
TRUSTEE_HOSTvariable by running the following command:$ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \ -o jsonpath={.spec.host})Verify the route by running the following command:
$ echo $TRUSTEE_HOSTExample output
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
3.15. Creating the authentication secret Copy linkLink copied to clipboard!
You create the authentication secret for Red Hat build of Trustee.
Procedure
Create a private key by running the following command:
$ openssl genpkey -algorithm ed25519 > privateKeyCreate a public key by running the following command:
$ openssl pkey -in privateKey -pubout -out publicKeyCreate a secret by running the following command:
$ oc create secret generic kbs-auth-public-key \ --from-file=publicKey -n trustee-operator-systemVerify the secret by running the following command:
$ oc get secret -n trustee-operator-system
3.16. Creating the KbsConfig custom resource Copy linkLink copied to clipboard!
Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.
Procedure
Create a
kbsconfig-cr.yamlmanifest file:apiVersion: confidentialcontainers.org/v1alpha1 kind: KbsConfig metadata: labels: app.kubernetes.io/name: kbsconfig app.kubernetes.io/instance: kbsconfig app.kubernetes.io/part-of: trustee-operator app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: trustee-operator name: kbsconfig namespace: trustee-operator-system spec: kbsConfigMapName: kbs-config-cm kbsAuthSecretName: kbs-auth-public-key kbsDeploymentType: AllInOneDeployment kbsRvpsRefValuesConfigMapName: rvps-reference-values kbsSecretResources: - attestation-status - <security-policy-name> kbsResourcePolicyConfigMapName: resource-policy kbsHttpsKeySecretName: kbs-https-key kbsHttpsCertSecretName: kbs-https-certificate kbsAttestationCertSecretName: attestation-cert kbsAttestationKeySecretName: attestation-key # tdxConfigSpec: # kbsTdxConfigMapName: tdx-config # kbsServiceType: <service_type>-
kbsSecretResources: Specify thetypevalue of the container image signature verification secret if you created the secret, for example,img-sig. -
Uncomment
tdxConfigSpec.kbsTdxConfigMapName: tdx-configfor Intel Trust Domain Extensions. -
Uncomment
kbsServiceType: <service_type>if you create a service type, other than the defaultClusterIPservice, to expose applications within the cluster external traffic. You can specifyNodePort,LoadBalancer, orExternalName.
-
Create the
KbsConfigCR by running the following command:$ oc create -f kbsconfig-cr.yaml
3.17. Updating the RVPS config map Copy linkLink copied to clipboard!
You update the Reference Value Provider Service (RVPS) config map with expected measurements, including the Platform Configuration Register (PCR) 8 value, for the trusted execution environment. Red Hat build of Trustee uses these measurements to verify the attestation evidence.
The workload cluster administrator calculates the PCR8 value by performing a SHA256 hash on a configuration or policy file such as initdata:
$ hash=$(sha256sum <config_file> | cut -d' ' -f1)
$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1)
Prerequisites
- PCR8 value, expiration, and algorithm, created by the workload cluster administrator
Procedure
Create an
rvps-configmap-update.yamlmanifest file:apiVersion: v1 kind: ConfigMap metadata: name: rvps-reference-values namespace: trustee-operator-system data: reference-values.json: | [ { "name": "svn", "expiration": "2027-01-01T00:00:00Z", "value" : 1 }, { "name": "major_version", "expiration": "2027-01-01T00:00:00Z", "value" : 1 }, { "name": "minimum_minor_version", "expiration": "2027-01-01T00:00:00Z", "value" : 4 } ]NoteDo not use this configuration example in a production environment.
The
"value"can be any JSON type (string, number, boolean, array, object). The JSON type must be combined with the operand used in theattestation-policy. See the following examples for valid"value"types:If the attestation rule is:
input.sample.platform_version.major == data.reference.major_versionThe
==operand expects to match an integer type in the reference values:{ "name": "major_version", "expiration": "2027-01-01T00:00:00Z", "value" : 1 }If the attestation rule is:
input.sample.svn in data.reference.svnThe
inoperand expects to match an array type in the reference values:{ "name": "svn", "expiration": "2027-01-01T00:00:00Z", "value" : [ 1 ] }
Update the RVPS config map by running the following command:
$ oc apply -f rvps-configmap-update.yaml
3.18. Verifying the configuration Copy linkLink copied to clipboard!
You verify the Red Hat build of Trustee configuration by checking its pods and logs.
Procedure
Set the default project by running the following command:
$ oc project trustee-operator-systemCheck the pods by running the following command:
$ oc get pods -n trustee-operator-systemExample output
NAME READY STATUS RESTARTS AGE trustee-deployment-8585f98449-9bbgl 1/1 Running 0 22m trustee-operator-controller-manager-5fbd44cd97-55dlh 2/2 Running 0 59mSet the
POD_NAMEenvironmental variable by running the following command:$ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)Check the pod logs by running the following command:
$ oc logs -n trustee-operator-system $POD_NAMEExample output
[2024-05-30T13:44:24Z INFO kbs] Using config file /etc/kbs-config/kbs-config.json [2024-05-30T13:44:24Z WARN attestation_service::rvps] No RVPS address provided and will launch a built-in rvps [2024-05-30T13:44:24Z INFO attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert [2024-05-30T13:44:24Z INFO api_server] Starting HTTPS server at [0.0.0.0:8080] [2024-05-30T13:44:24Z INFO actix_server::builder] starting 12 workers [2024-05-30T13:44:24Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime