Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 4. Deploying on Azure

You can deploy OpenShift sandboxed containers and Confidential Containers on Microsoft Azure Cloud Computing Services.

Cluster requirements

  • You have installed Red Hat OpenShift Container Platform 4.14 or later on the cluster where you are installing the OpenShift sandboxed containers Operator.
  • Your cluster has at least one worker node.

4.1. Peer pod resource requirements
Copy link

You must ensure that your cluster has sufficient resources.

Peer pod virtual machines (VMs) require resources in two locations:

  • The worker node. The worker node stores metadata, Kata shim resources (containerd-shim-kata-v2), remote-hypervisor resources (cloud-api-adaptor), and the tunnel setup between the worker nodes and the peer pod VM.
  • The cloud instance. This is the actual peer pod VM running in the cloud.

The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote) definition used for creating peer pods.

The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit attribute in the peerpodConfig custom resource (CR).

The peerpodConfig CR, named peerpodconfig-openshift, is created when you create the kataConfig CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator namespace.

The following peerpodConfig CR example displays the default spec values: 

apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
  name: peerpodconfig-openshift
  namespace: openshift-sandboxed-containers-operator
spec:
  cloudSecretName: peer-pods-secret
  configMapName: peer-pods-cm
  limit: "10"
1 

  nodeSelector:
    node-role.kubernetes.io/kata-oc: ""
Copy to Clipboard Toggle word wrap
1
The default limit is 10 VMs per node.

The extended resource is named kata.peerpods.io/vm, and enables the Kubernetes scheduler to handle capacity tracking and accounting.

You can edit the limit per node based on the requirements for your environment after you install the OpenShift sandboxed containers Operator.

A mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available.

The mutating webhook modifies a Kubernetes pod as follows:

  • The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIME_CLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIME_CLASS, the webhook exits without modifying the pod.

  • If the RuntimeClassName values match, the webhook makes the following changes to the pod spec:

    1. The webhook removes every resource specification from the resources field of all containers and init containers in the pod.
    2. The webhook adds the extended resource (kata.peerpods.io/vm) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes.
Note

The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.

As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces.

You can deploy OpenShift sandboxed containers on Azure by using the OpenShift Container Platform web console to perform the following tasks:

  1. Install the OpenShift sandboxed containers Operator.
  2. Create the peer pods secret.
  3. Create the peer pods config map.
  4. Create the Azure secret.
  5. Create the KataConfig custom resource.
  6. Configure the OpenShift sandboxed containers workload objects.

You can install the OpenShift sandboxed containers Operator by using the OpenShift Container Platform web console.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. In the web console, navigate to Operators OperatorHub.
  2. In the Filter by keyword field, type OpenShift sandboxed containers.
  3. Select the OpenShift sandboxed containers Operator tile and click Install.
  4. On the Install Operator page, select stable from the list of available Update Channel options.

  5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

    Note

    Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

  6. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
  7. Click Install.
  8. Navigate to Operators Installed Operators to verify that the Operator is installed.

4.2.2. Creating the peer pods secret
Copy link

When the peer pods secret is empty and the Cloud Credential Operator (CCO) is installed, the OpenShift sandboxed containers Operator uses the CCO to retrieve the secret. If you have uninstalled the CCO, you must create the peer pods secret for OpenShift sandboxed containers manually or the peer pods will fail to operate.

The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances.

By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.

Prerequisites

  • You have installed and configured the Azure CLI tool.

Procedure

  1. Retrieve the Azure subscription ID by running the following command: 

    $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" \
  -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
    Copy to Clipboard Toggle word wrap

  2. Generate the RBAC content by running the following command: 

    $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
  --query "{ client_id: appId, client_secret: password, tenant_id: tenant }"
    Copy to Clipboard Toggle word wrap

    Example output

    {
  "client_id": `AZURE_CLIENT_ID`,
  "client_secret": `AZURE_CLIENT_SECRET`,
  "tenant_id": `AZURE_TENANT_ID`
}
    Copy to Clipboard Toggle word wrap

  3. Record the RBAC output to use in the secret object.
  4. In the OpenShift Container Platform web console, navigate to Operators Installed Operators.
  5. Click the OpenShift sandboxed containers Operator tile.
  6. Click the Import icon (+) on the top right corner.

  7. In the Import YAML window, paste the following YAML manifest: 

    apiVersion: v1
kind: Secret
metadata:
  name: peer-pods-secret
  namespace: openshift-sandboxed-containers-operator
type: Opaque
stringData:
  AZURE_CLIENT_ID: "<azure_client_id>"
    1 
    
  AZURE_CLIENT_SECRET: "<azure_client_secret>"
    2 
    
  AZURE_TENANT_ID: "<azure_tenant_id>"
    3 
    
  AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>"
    4
    Copy to Clipboard Toggle word wrap
    1
    Specify the AZURE_CLIENT_ID value.
    2
    Specify the AZURE_CLIENT_SECRET value.
    3
    Specify the AZURE_TENANT_ID value.
    4
    Specify the AZURE_SUBSCRIPTION_ID value.
  8. Click Save to apply the changes.
  9. Navigate to Workloads Secrets to verify the peer pods secret.

4.2.3. Creating the peer pods config map
Copy link

You must create the peer pods config map for OpenShift sandboxed containers.

Procedure

  1. Obtain the following values from your Azure instance:

    1. Retrieve and record the Azure resource group: 

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
      Copy to Clipboard Toggle word wrap

    2. Retrieve and record the Azure VNet name: 

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

    3. Retrieve and record the Azure subnet ID: 

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
      Copy to Clipboard Toggle word wrap

    4. Retrieve and record the Azure network security group (NSG) ID: 

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
      Copy to Clipboard Toggle word wrap

    5. Retrieve and record the Azure region: 

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
      Copy to Clipboard Toggle word wrap
  2. In the OpenShift Container Platform web console, navigate to Operators Installed Operators.
  3. Select the OpenShift sandboxed containers Operator from the list of operators.
  4. Click the Import icon (+) in the top right corner.

  5. In the Import YAML window, paste the following YAML manifest: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "azure"
  VXLAN_PORT: "9000"
  AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
    1 
    
  AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5"
    2 
    
  AZURE_SUBNET_ID: "<azure_subnet_id>"
    3 
    
  AZURE_NSG_ID: "<azure_nsg_id>"
    4 
    
  PROXY_TIMEOUT: "5m"
  AZURE_IMAGE_ID: "<azure_image_id>"
    5 
    
  AZURE_REGION: "<azure_region>"
    6 
    
  AZURE_RESOURCE_GROUP: "<azure_resource_group>"
    7 
    
  DISABLECVM: "true"
    Copy to Clipboard Toggle word wrap
    1
    The "Standard_B2als_v2" instance size is the default value if an instance size is not defined in the workload.
    2
    Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
    3
    Specify the AZURE_SUBNET_ID value that you retrieved.
    4
    Specify the AZURE_NSG_ID value that you retrieved.
    5
    Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.
    6
    Specify the AZURE_REGION value you retrieved.
    7
    Specify the AZURE_RESOURCE_GROUP value you retrieved.
  6. Click Save to apply the changes.
  7. Navigate to Workloads ConfigMaps to view the new config map.

4.2.4. Creating the Azure secret
Copy link

You must create the SSH key secret, which is required by the Azure virtual machine (VM) creation API. Azure only requires the SSH public key. Confidential Containers disables SSH in VMs, so the keys have no effect in the VMs.

Procedure

  1. Generate an SSH key pair by running the following command: 

    $ ssh-keygen -f ./id_rsa -N ""
    Copy to Clipboard Toggle word wrap
  2. In the OpenShift Container Platform web console, navigate to Workloads Secrets.
  3. On the Secrets page, verify that you are in the openshift-sandboxed-containers-operator project.
  4. Click Create and select Key/value secret.
  5. In the Secret name field, enter ssh-key-secret.
  6. In the Key field, enter id_rsa.pub.
  7. In the Value field, paste your public SSH key.
  8. Click Create.

  9. Delete the SSH keys you created: 

    $ shred --remove id_rsa.pub id_rsa
    Copy to Clipboard Toggle word wrap

4.2.5. Creating the KataConfig custom resource
Copy link

You must create the KataConfig custom resource (CR) to install kata-remote as a RuntimeClass on your worker nodes.

The kata-remote runtime class is installed on all worker nodes by default. If you want to install kata-remote on specific nodes, you can add labels to those nodes and then define the label in the KataConfig CR.

OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors might increase the reboot time:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • Optional: You have installed the Node Feature Discovery Operator if you want to enable node eligibility checks.

Procedure

  1. In the OpenShift Container Platform web console, navigate to Operators Installed Operators.
  2. Select the OpenShift sandboxed containers Operator.
  3. On the KataConfig tab, click Create KataConfig.

  4. Enter the following details:

    • Name: Optional: The default name is example-kataconfig.
    • Labels: Optional: Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
    • enablePeerPods: Select for public cloud, IBM Z®, and IBM® LinuxONE deployments.

    • kataConfigPoolSelector. Optional: To install kata-remote on selected nodes, add a match expression for the labels on the selected nodes:

      1. Expand the kataConfigPoolSelector area.
      2. In the kataConfigPoolSelector area, expand matchExpressions. This is a list of label selector requirements.
      3. Click Add matchExpressions.
      4. In the Key field, enter the label key the selector applies to.
      5. In the Operator field, enter the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
      6. Expand the Values area and then click Add value.
      7. In the Value field, enter true or false for key label value.
    • logLevel: Define the level of log data retrieved for nodes with the kata-remote runtime class.

  5. Click Create. The KataConfig CR is created and installs the kata-remote runtime class on the worker nodes.

    Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

Verification

  1. On the KataConfig tab, click the KataConfig CR to view its details.

  2. Click the YAML tab to view the status stanza.

    The status stanza contains the conditions and kataNodes keys. The value of status.kataNodes is an array of nodes, each of which lists nodes in a particular state of kata-remote installation. A message appears each time there is an update.

  3. Click Reload to refresh the YAML.

    When all workers in the status.kataNodes array display the values installed and conditions.InProgress: False with no specified reason, the kata-remote is installed on the cluster.

Additional resources
Verifying the pod VM image

After kata-remote is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.

Procedure

  1. Navigate to Workloads ConfigMaps.
  2. Click the provider config map to view its details.
  3. Click the YAML tab.

  4. Check the status stanza of the YAML file.

    If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

  1. Retrieve the events log by running the following command: 

    $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
    Copy to Clipboard Toggle word wrap

  2. Retrieve the job log by running the following command: 

    $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
    Copy to Clipboard Toggle word wrap

If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.

4.2.6. Configuring workload objects
Copy link

You must configure OpenShift sandboxed containers workload objects by setting kata-remote as the runtime class for the following pod-templated objects:

  • Pod objects
  • ReplicaSet objects
  • ReplicationController objects
  • StatefulSet objects
  • Deployment objects
  • DeploymentConfig objects
Important

Do not deploy workloads in an Operator namespace. Create a dedicated namespace for these resources.

You can define whether the workload should be deployed using the default instance size, which you defined in the config map, by adding an annotation to the YAML file.

If you do not want to define the instance size manually, you can add an annotation to use an automatic instance size, based on the memory available.

Prerequisites

  • You have created the KataConfig custom resource (CR).

Procedure

  1. In the OpenShift Container Platform web console, navigate to Workloads workload type, for example, Pods.
  2. On the workload type page, click an object to view its details.
  3. Click the YAML tab.

  4. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example: 

    apiVersion: v1
kind: <object>
# ...
spec:
  runtimeClassName: kata-remote
# ...
    Copy to Clipboard Toggle word wrap

  5. Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:

    • To use a manually defined instance size, add the following annotation: 

      apiVersion: v1
kind: <object>
metadata:
  annotations:
    io.katacontainers.config.hypervisor.machine_type: "Standard_B2als_v2"
      1 
      
# ...
      Copy to Clipboard Toggle word wrap
      1
      Specify the instance size that you defined in the config map.

    • To use an automatic instance size, add the following annotations: 

      apiVersion: v1
kind: <Pod>
metadata:
  annotations:
    io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
    io.katacontainers.config.hypervisor.default_memory: <memory>
# ...
      Copy to Clipboard Toggle word wrap

      Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.

  6. Click Save to apply the changes.

    OpenShift Container Platform creates the workload object and begins scheduling it.

Verification

  • Inspect the spec.runtimeClassName field of a pod-templated object. If the value is kata-remote, then the workload is running on OpenShift sandboxed containers, using peer pods.

You can deploy OpenShift sandboxed containers on Azure by using the command line interface (CLI) to perform the following tasks:

  1. Install the OpenShift sandboxed containers Operator.
  2. Optional: Change the number of virtual machines running on each worker node.
  3. Create the peer pods secret.
  4. Create the peer pods config map.
  5. Create the Azure secret.
  6. Create the KataConfig custom resource.
  7. Configure the OpenShift sandboxed containers workload objects.

You can install the OpenShift sandboxed containers Operator by using the CLI.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create an osc-namespace.yaml manifest file: 

    apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

  2. Create the namespace by running the following command: 

    $ oc apply -f osc-namespace.yaml
    Copy to Clipboard Toggle word wrap

  3. Create an osc-operatorgroup.yaml manifest file: 

    apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

  4. Create the operator group by running the following command: 

    $ oc apply -f osc-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap

  5. Create an osc-subscription.yaml manifest file: 

    apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.7.0
    Copy to Clipboard Toggle word wrap

  6. Create the subscription by running the following command: 

    $ oc apply -f osc-subscription.yaml
    Copy to Clipboard Toggle word wrap

  7. Verify that the Operator is correctly installed by running the following command: 

    $ oc get csv -n openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command: 

    $ watch oc get csv -n openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.7.0    1.6.0        Succeeded
    Copy to Clipboard Toggle word wrap

You can change the limit of peer pod virtual machines (VMs) per node by editing the peerpodConfig custom resource (CR).

Procedure

  1. Check the current limit by running the following command: 

    $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
-o jsonpath='{.spec.limit}{"\n"}'
    Copy to Clipboard Toggle word wrap

  2. Modify the limit attribute of the peerpodConfig CR by running the following command: 

    $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
--type merge --patch '{"spec":{"limit":"<value>"}}'
    1
    Copy to Clipboard Toggle word wrap
    1
    Replace <value> with the limit you want to define.

4.3.3. Creating the peer pods secret
Copy link

When the peer pods secret is empty and the Cloud Credential Operator (CCO) is installed, the OpenShift sandboxed containers Operator uses the CCO to retrieve the secret. If you have uninstalled the CCO, you must create the peer pods secret for OpenShift sandboxed containers manually or the peer pods will fail to operate.

The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances.

By default, the OpenShift sandboxed containers Operator creates the secret based on the credentials used to create the cluster. However, you can manually create a secret that uses different credentials.

Prerequisites

  • You have installed and configured the Azure CLI tool.

Procedure

  1. Retrieve the Azure subscription ID by running the following command: 

    $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" \
  -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
    Copy to Clipboard Toggle word wrap

  2. Generate the RBAC content by running the following command: 

    $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
  --query "{ client_id: appId, client_secret: password, tenant_id: tenant }"
    Copy to Clipboard Toggle word wrap

    Example output

    {
  "client_id": `AZURE_CLIENT_ID`,
  "client_secret": `AZURE_CLIENT_SECRET`,
  "tenant_id": `AZURE_TENANT_ID`
}
    Copy to Clipboard Toggle word wrap

  3. Record the RBAC output to use in the secret object.

  4. Create a peer-pods-secret.yaml manifest file according to the following example: 

    apiVersion: v1
kind: Secret
metadata:
  name: peer-pods-secret
  namespace: openshift-sandboxed-containers-operator
type: Opaque
stringData:
  AZURE_CLIENT_ID: "<azure_client_id>"
    1 
    
  AZURE_CLIENT_SECRET: "<azure_client_secret>"
    2 
    
  AZURE_TENANT_ID: "<azure_tenant_id>"
    3 
    
  AZURE_SUBSCRIPTION_ID: "<azure_subscription_id>"
    4
    Copy to Clipboard Toggle word wrap
    1
    Specify the AZURE_CLIENT_ID value.
    2
    Specify the AZURE_CLIENT_SECRET value.
    3
    Specify the AZURE_TENANT_ID value.
    4
    Specify the AZURE_SUBSCRIPTION_ID value.

  5. Create the secret by running the following command: 

    $ oc apply -f peer-pods-secret.yaml
    Copy to Clipboard Toggle word wrap

4.3.4. Creating the peer pods config map
Copy link

You must create the peer pods config map for OpenShift sandboxed containers.

Procedure

  1. Obtain the following values from your Azure instance:

    1. Retrieve and record the Azure resource group: 

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
      Copy to Clipboard Toggle word wrap

    2. Retrieve and record the Azure VNet name: 

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

    3. Retrieve and record the Azure subnet ID: 

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
      Copy to Clipboard Toggle word wrap

    4. Retrieve and record the Azure network security group (NSG) ID: 

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
      Copy to Clipboard Toggle word wrap

    5. Retrieve and record the Azure region: 

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
      Copy to Clipboard Toggle word wrap

  2. Create a peer-pods-cm.yaml manifest file according to the following example: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "azure"
  VXLAN_PORT: "9000"
  AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
    1 
    
  AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5"
    2 
    
  AZURE_SUBNET_ID: "<azure_subnet_id>"
    3 
    
  AZURE_NSG_ID: "<azure_nsg_id>"
    4 
    
  PROXY_TIMEOUT: "5m"
  AZURE_IMAGE_ID: "<azure_image_id>"
    5 
    
  AZURE_REGION: "<azure_region>"
    6 
    
  AZURE_RESOURCE_GROUP: "<azure_resource_group>"
    7 
    
  DISABLECVM: "true"
    Copy to Clipboard Toggle word wrap
    1
    The "Standard_B2als_v2" instance size is the default value if an instance size is not defined in the workload.
    2
    Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
    3
    Specify the AZURE_SUBNET_ID value that you retrieved.
    4
    Specify the AZURE_NSG_ID value that you retrieved.
    5
    Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.
    6
    Specify the AZURE_REGION value you retrieved.
    7
    Specify the AZURE_RESOURCE_GROUP value you retrieved.

  3. Create the config map by running the following command: 

    $ oc apply -f peer-pods-cm.yaml
    Copy to Clipboard Toggle word wrap

4.3.5. Creating the Azure secret
Copy link

You must create the SSH key secret, which is required by the Azure virtual machine (VM) creation API. Azure only requires the SSH public key. Confidential Containers disables SSH in VMs, so the keys have no effect in the VMs.

Procedure

  1. Generate an SSH key pair by running the following command: 

    $ ssh-keygen -f ./id_rsa -N ""
    Copy to Clipboard Toggle word wrap

  2. Create the Secret object by running the following command: 

    $ oc create secret generic ssh-key-secret \
  -n openshift-sandboxed-containers-operator \
  --from-file=id_rsa.pub=./id_rsa.pub \
  --from-file=id_rsa=./id_rsa
    Copy to Clipboard Toggle word wrap

  3. Delete the SSH keys you created: 

    $ shred --remove id_rsa.pub id_rsa
    Copy to Clipboard Toggle word wrap

4.3.6. Creating the KataConfig custom resource
Copy link

You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes.

Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following:

  • Create a RuntimeClass CR named kata-remote with a default configuration. This enables users to configure workloads to use kata-remote as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime.

OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create an example-kataconfig.yaml manifest file according to the following example: 

    apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
    1
    Copy to Clipboard Toggle word wrap
    1
    Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

  2. Create the KataConfig CR by running the following command: 

    $ oc apply -f example-kataconfig.yaml
    Copy to Clipboard Toggle word wrap

    The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

    Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

  3. Monitor the installation progress by running the following command: 

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
    Copy to Clipboard Toggle word wrap

    When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

  4. Verify the daemon set by running the following command: 

    $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon
    Copy to Clipboard Toggle word wrap

  5. Verify the runtime classes by running the following command: 

    $ oc get runtimeclass
    Copy to Clipboard Toggle word wrap

    Example output

    NAME             HANDLER          AGE
kata             kata             152m
kata-remote      kata-remote      152m
    Copy to Clipboard Toggle word wrap

Verifying the pod VM image

After kata-remote is installed on your cluster, the OpenShift sandboxed containers Operator creates a pod VM image, which is used to create peer pods. This process can take a long time because the image is created on the cloud instance. You can verify that the pod VM image was created successfully by checking the config map that you created for the cloud provider.

Procedure

  1. Obtain the config map you created for the peer pods: 

    $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml
    Copy to Clipboard Toggle word wrap

  2. Check the status stanza of the YAML file.

    If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

  1. Retrieve the events log by running the following command: 

    $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation
    Copy to Clipboard Toggle word wrap

  2. Retrieve the job log by running the following command: 

    $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation
    Copy to Clipboard Toggle word wrap

If you cannot resolve the issue, submit a Red Hat Support case and attach the output of both logs.

4.3.7. Configuring workload objects
Copy link

You must configure OpenShift sandboxed containers workload objects by setting kata-remote as the runtime class for the following pod-templated objects:

  • Pod objects
  • ReplicaSet objects
  • ReplicationController objects
  • StatefulSet objects
  • Deployment objects
  • DeploymentConfig objects
Important

Do not deploy workloads in an Operator namespace. Create a dedicated namespace for these resources.

You can define whether the workload should be deployed using the default instance size, which you defined in the config map, by adding an annotation to the YAML file.

If you do not want to define the instance size manually, you can add an annotation to use an automatic instance size, based on the memory available.

Prerequisites

  • You have created the KataConfig custom resource (CR).

Procedure

  1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example: 

    apiVersion: v1
kind: <object>
# ...
spec:
  runtimeClassName: kata-remote
# ...
    Copy to Clipboard Toggle word wrap

  2. Add an annotation to the pod-templated object to use a manually defined instance size or an automatic instance size:

    • To use a manually defined instance size, add the following annotation: 

      apiVersion: v1
kind: <object>
metadata:
  annotations:
    io.katacontainers.config.hypervisor.machine_type: "Standard_B2als_v2"
      1 
      
# ...
      Copy to Clipboard Toggle word wrap
      1
      Specify the instance size that you defined in the config map.

    • To use an automatic instance size, add the following annotations: 

      apiVersion: v1
kind: <Pod>
metadata:
  annotations:
    io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
    io.katacontainers.config.hypervisor.default_memory: <memory>
# ...
      Copy to Clipboard Toggle word wrap

      Define the amount of memory available for the workload to use. The workload will run on an automatic instance size based on the amount of memory available.

  3. Apply the changes to the workload object by running the following command: 

    $ oc apply -f <object.yaml>
    Copy to Clipboard Toggle word wrap

    OpenShift Container Platform creates the workload object and begins scheduling it.

Verification

  • Inspect the spec.runtimeClassName field of a pod-templated object. If the value is kata-remote, then the workload is running on OpenShift sandboxed containers, using peer pods.

4.4. Deploying Confidential Containers on Azure
Copy link

You can deploy Confidential Containers on Microsoft Azure Cloud Computing Services after you deploy OpenShift sandboxed containers.

Important

Confidential Containers on Azure is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Cluster requirements

  • You have installed Red Hat OpenShift Container Platform 4.15 or later on the cluster where you are installing the Confidential compute attestation Operator.

You deploy Confidential Containers by performing the following steps:

  1. Install the Confidential compute attestation Operator.
  2. Create the route for Trustee.
  3. Enable the Confidential Containers feature gate.
  4. Update the peer pods config map.
  5. Delete the KataConfig custom resource (CR).
  6. Re-create the KataConfig CR.
  7. Create the Trustee authentication secret.
  8. Create the Trustee config map.

  9. Configure attestation policies:

    1. Create reference values.
    2. Create secrets for attested clients.
    3. Create the resource access policy.
    4. Optional: Create an attestation policy that overrides the default policy.
    5. If your TEE is Intel Trust Domain Extensions, configure the Provisioning Certificate Caching Service.
  10. Create the KbsConfig CR.
  11. Verify the attestation process.

You can install the Confidential compute attestation Operator on Azure by using the CLI.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create a trustee-namespace.yaml manifest file: 

    apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system
    Copy to Clipboard Toggle word wrap

  2. Create the trustee-operator-system namespace by running the following command: 

    $ oc apply -f trustee-namespace.yaml
    Copy to Clipboard Toggle word wrap

  3. Create a trustee-operatorgroup.yaml manifest file: 

    apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system
    Copy to Clipboard Toggle word wrap

  4. Create the operator group by running the following command: 

    $ oc apply -f trustee-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap

  5. Create a trustee-subscription.yaml manifest file: 

    apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: trustee-operator.v0.1.0
    Copy to Clipboard Toggle word wrap

  6. Create the subscription by running the following command: 

    $ oc apply -f trustee-subscription.yaml
    Copy to Clipboard Toggle word wrap

  7. Verify that the Operator is correctly installed by running the following command: 

    $ oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command: 

    $ watch oc get csv -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                      DISPLAY                        PHASE
trustee-operator.v0.1.0   Trustee Operator  0.1.0        Succeeded
    Copy to Clipboard Toggle word wrap

4.4.2. Creating the route for Trustee
Copy link

You can create a secure route with edge TLS termination for Trustee. External ingress traffic reaches the router pods as HTTPS and passes on to the Trustee pods as HTTP.

Prerequisites

  • You have enabled the Confidential Containers feature gate.
  • You have installed the Confidential compute attestation Operator.

Procedure

  1. Create an edge route by running the following command: 

    $ oc create route edge --service=kbs-service --port kbs-port \
  -n trustee-operator-system
    Copy to Clipboard Toggle word wrap
    Note

    Note: Currently, only a route with a valid CA-signed certificate is supported. You cannot use a route with self-signed certificate.

  2. Set the TRUSTEE_HOST variable by running the following command: 

    $ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})
    Copy to Clipboard Toggle word wrap

  3. Verify the route by running the following command: 

    $ echo $TRUSTEE_HOST
    Copy to Clipboard Toggle word wrap

    Example output

    kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
    Copy to Clipboard Toggle word wrap

    Record this value for the peer pods config map.

You must enable the Confidential Containers feature gate.

Procedure

  1. Create a cc-feature-gate.yaml manifest file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
    Copy to Clipboard Toggle word wrap

  2. Create the config map by running the following command: 

    $ oc apply -f cc-feature-gate.yaml
    Copy to Clipboard Toggle word wrap

4.4.4. Updating the peer pods config map
Copy link

You must update the peer pods config map for Confidential Containers.

Note

Set Secure Boot to true to enable it by default. The default value is false, which presents a security risk.

Procedure

  1. Obtain the following values from your Azure instance:

    1. Retrieve and record the Azure resource group: 

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
      Copy to Clipboard Toggle word wrap

    2. Retrieve and record the Azure VNet name: 

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

    3. Retrieve and record the Azure subnet ID: 

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
      Copy to Clipboard Toggle word wrap

    4. Retrieve and record the Azure network security group (NSG) ID: 

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
      Copy to Clipboard Toggle word wrap

    5. Retrieve and record the Azure region: 

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
      Copy to Clipboard Toggle word wrap

  2. Create a peer-pods-cm.yaml manifest file according to the following example: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "azure"
  VXLAN_PORT: "9000"
  AZURE_INSTANCE_SIZE: "Standard_DC2as_v5"
    1 
    
  AZURE_INSTANCE_SIZES: "Standard_DC2as_v5, Standard_DC4as_v5, Standard_DC8as_v5"
    2 
    
  AZURE_SUBNET_ID: "<azure_subnet_id>"
    3 
    
  AZURE_NSG_ID: "<azure_nsg_id>"
    4 
    
  PROXY_TIMEOUT: "5m"
  AZURE_IMAGE_ID: "<azure_image_id>"
    5 
    
  AZURE_REGION: "<azure_region>"
    6 
    
  AZURE_RESOURCE_GROUP: "<azure_resource_group>"
    7 
    
  DISABLECVM: "false"
  AA_KBC_PARAMS: "cc_kbc::https://${TRUSTEE_HOST}"
    8 
    
  ENABLE_SECURE_BOOT: "true"
    9
    Copy to Clipboard Toggle word wrap
    1
    The "Standard_DC2as_v5" value is the default if an instance size is not defined in the workload. Ensure the instance type supports the trusted environment. The default "Standard_DC2as_v5" value is for AMD SEV-SNP. If your TEE is Intel TDX, specify Standard_EC4eds_v5.
    2
    Lists all of the instance sizes you can specify when creating the pod. This allows you to define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads. For Intel TDX, specify "Standard_EC4eds_v5, Standard_EC8eds_v5, Standard_EC16eds_v5".
    3
    Specify the AZURE_SUBNET_ID value that you retrieved.
    4
    Specify the AZURE_NSG_ID value that you retrieved.
    5
    Optional: By default, this value is populated when you run the KataConfig CR, using an Azure image ID based on your cluster credentials. If you create your own Azure image, specify the correct image ID.
    6
    Specify the AZURE_REGION value you retrieved.
    7
    Specify the AZURE_RESOURCE_GROUP value you retrieved.
    8
    Specify the host name of the Trustee route.
    9
    Specify true to enable Secure Boot by default.

  3. Create the config map by running the following command: 

    $ oc apply -f peer-pods-cm.yaml
    Copy to Clipboard Toggle word wrap

  4. Restart the peerpodconfig-ctrl-caa-daemon daemon set by running the following command: 

    $ oc set env ds/peerpodconfig-ctrl-caa-daemon \
  -n openshift-sandboxed-containers-operator REBOOT="$(date)"
    Copy to Clipboard Toggle word wrap

4.4.5. Deleting the KataConfig custom resource
Copy link

You can delete the KataConfig custom resource (CR) by using the command line.

Deleting the KataConfig CR removes the runtime and its related resources from your cluster.

Important

Deleting the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Delete the KataConfig CR by running the following command: 

    $ oc delete kataconfig example-kataconfig
    Copy to Clipboard Toggle word wrap

    The OpenShift sandboxed containers Operator removes all resources that were initially created to enable the runtime on your cluster.

    Important

    When you delete the KataConfig CR, the CLI stops responding until all worker nodes reboot. You must wait for the deletion process to complete before performing the verification.

  2. Verify that the custom resource was deleted by running the following command: 

    $ oc get kataconfig example-kataconfig
    Copy to Clipboard Toggle word wrap

    Example output

    No example-kataconfig instances exist
    Copy to Clipboard Toggle word wrap

Important

When uninstalling OpenShift sandboxed containers deployed using a cloud provider, you must delete all of the pods. Any remaining pod resources might result in an unexpected bill from your cloud provider.

4.4.6. Re-creating the KataConfig custom resource
Copy link

You must re-create the KataConfig custom resource (CR) for Confidential Containers.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create an example-kataconfig.yaml manifest file according to the following example: 

    apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
    1
    Copy to Clipboard Toggle word wrap
    1
    Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true'.

  2. Create the KataConfig CR by running the following command: 

    $ oc apply -f example-kataconfig.yaml
    Copy to Clipboard Toggle word wrap

    The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

    Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

  3. Monitor the installation progress by running the following command: 

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
    Copy to Clipboard Toggle word wrap

    When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

  4. Verify the daemon set by running the following command: 

    $ oc get -n openshift-sandboxed-containers-operator ds/peerpodconfig-ctrl-caa-daemon
    Copy to Clipboard Toggle word wrap

  5. Verify the runtime classes by running the following command: 

    $ oc get runtimeclass
    Copy to Clipboard Toggle word wrap

    Example output

    NAME             HANDLER          AGE
kata             kata             152m
kata-remote      kata-remote      152m
    Copy to Clipboard Toggle word wrap

4.4.7. Creating the Trustee authentication secret
Copy link

You must create the authentication secret for Trustee.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create a private key by running the following command: 

    $ openssl genpkey -algorithm ed25519 > privateKey
    Copy to Clipboard Toggle word wrap

  2. Create a public key by running the following command: 

    $ openssl pkey -in privateKey -pubout -out publicKey
    Copy to Clipboard Toggle word wrap

  3. Create a secret by running the following command: 

    $ oc create secret generic kbs-auth-public-key --from-file=publicKey -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

  4. Verify the secret by running the following command: 

    $ oc get secret -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

4.4.8. Creating the Trustee config map
Copy link

You must create the config map to configure the Trustee server.

Prerequisites

  • You have created a route for Trustee.

Procedure

  1. Create a kbs-config-cm.yaml manifest file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.json: |
    {
      "insecure_http" : true,
      "sockets": ["0.0.0.0:8080"],
      "auth_public_key": "/etc/auth-secret/publicKey",
      "attestation_token_config": {
        "attestation_token_type": "CoCo"
      },
      "repository_config": {
        "type": "LocalFs",
        "dir_path": "/opt/confidential-containers/kbs/repository"
      },
      "as_config": {
        "work_dir": "/opt/confidential-containers/attestation-service",
        "policy_engine": "opa",
        "attestation_token_broker": "Simple",
          "attestation_token_config": {
          "duration_min": 5
          },
        "rvps_config": {
          "store_type": "LocalJson",
          "store_config": {
            "file_path": "/opt/confidential-containers/rvps/reference-values/reference-values.json"
          }
         }
      },
      "policy_engine_config": {
        "policy_path": "/opt/confidential-containers/opa/policy.rego"
      }
    }
    Copy to Clipboard Toggle word wrap

  2. Create the config map by running the following command: 

    $ oc apply -f kbs-config-cm.yaml
    Copy to Clipboard Toggle word wrap

4.4.9. Configuring attestation policies
Copy link

You can configure the following attestation policy settings:

Reference values

You can configure reference values for the Reference Value Provider Service (RVPS) by specifying the trusted digests of your hardware platform.

The client collects measurements from the running software, the Trusted Execution Environment (TEE) hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to the Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

Secrets for clients
You must create one or more secrets to share with attested clients.
Resource access policy

You must configure a policy for the Trustee policy engine to determine which resources to access.

Do not confuse the Trustee policy engine with the Attestation Service policy engine, which determines the validity of TEE evidence.

Attestation policy
Optional: You can overwrite the default attestation policy by creating your own attestation policy.
Provisioning Certificate Caching Service for TDX

If your TEE is Intel Trust Domain Extensions (TDX), you must configure the Provisioning Certificate Caching Service (PCCS). The PCCS retrieves Provisioning Certification Key (PCK) certificates and caches them in a local database.

Important

Do not use the public Intel PCCS service. Use a local caching service on-premise or on the public cloud.

Procedure

  1. Create an rvps-configmap.yaml manifest file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
    1 
    
    ]
    Copy to Clipboard Toggle word wrap
    1
    Specify the trusted digests for your hardware platform if required. Otherwise, leave it empty.

  2. Create the RVPS config map by running the following command: 

    $ oc apply -f rvps-configmap.yaml
    Copy to Clipboard Toggle word wrap

  3. Create one or more secrets to share with attested clients according to the following example: 

    $ oc create secret generic kbsres1 --from-literal key1=<res1val1> \
  --from-literal key2=<res1val2> -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    In this example, the kbsres1 secret has two entries (key1, key2), which the Trustee clients retrieve. You can add more secrets according to your requirements.

  4. Create a resourcepolicy-configmap.yaml manifest file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    1 
    
    package policy
    2 
    
    default allow = false
    allow {
      input["tee"] != "sample"
    }
    Copy to Clipboard Toggle word wrap
    1
    The name of the resource policy, policy.rego, must match the resource policy defined in the Trustee config map.
    2
    The resource policy follows the Open Policy Agent specification. This example allows the retrieval of all resources when the TEE is not the sample attester.

  5. Create the resource policy config map by running the following command: 

    $ oc apply -f resourcepolicy-configmap.yaml
    Copy to Clipboard Toggle word wrap

  6. Optional: Create an attestation-policy.yaml manifest file according to the following example: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default.rego: |
     package policy
    1 
    
     import future.keywords.every

     default allow = false

     allow {
        every k, v in input {
            judge_field(k, v)
        }
     }

     judge_field(input_key, input_value) {
        has_key(data.reference, input_key)
        reference_value := data.reference[input_key]
        match_value(reference_value, input_value)
     }

     judge_field(input_key, input_value) {
        not has_key(data.reference, input_key)
     }

     match_value(reference_value, input_value) {
        not is_array(reference_value)
        input_value == reference_value
     }

     match_value(reference_value, input_value) {
        is_array(reference_value)
        array_include(reference_value, input_value)
     }

     array_include(reference_value_array, input_value) {
        reference_value_array == []
     }

     array_include(reference_value_array, input_value) {
        reference_value_array != []
        some i
        reference_value_array[i] == input_value
     }

     has_key(m, k) {
        _ = m[k]
     }
    Copy to Clipboard Toggle word wrap
    1
    The attestation policy follows the Open Policy Agent specification. In this example, the attestation policy compares the claims provided in the attestation report to the reference values registered in the RVPS database. The attestation process is successful only if all the values match.

  7. Create the attestation policy config map by running the following command: 

    $ oc apply -f attestation-policy.yaml
    Copy to Clipboard Toggle word wrap

  8. If your TEE is Intel TDX, create a tdx-config.yaml manifest file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: tdx-config
  namespace: trustee-operator-system
data:
  sgx_default_qcnl.conf: | \
      {
        "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/",
        "pccs_url": "<pccs_url>"
    1 
    
      }
    Copy to Clipboard Toggle word wrap
    1
    Specify the PCCS URL, for example, https://localhost:8081/sgx/certification/v4/.

  9. Create the TDX config map by running the following command: 

    $ oc apply -f tdx-config.yaml
    Copy to Clipboard Toggle word wrap

4.4.10. Creating the KbsConfig custom resource
Copy link

You must create the KbsConfig custom resource (CR) to launch Trustee.

Then, you check the Trustee pods and pod logs to verify the configuration.

Procedure

  1. Create a kbsconfig-cr.yaml manifest file: 

    apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources: ["kbsres1"]
  kbsResourcePolicyConfigMapName: resource-policy
    Copy to Clipboard Toggle word wrap

  2. Create the KbsConfig CR by running the following command: 

    $ oc apply -f kbsconfig-cr.yaml
    Copy to Clipboard Toggle word wrap

Verification

  1. Set the default project by running the following command: 

    $ oc project trustee-operator-system
    Copy to Clipboard Toggle word wrap

  2. Check the pods by running the following command: 

    $ oc get pods -n trustee-operator-system
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m
    Copy to Clipboard Toggle word wrap

  3. Set the POD_NAME environmental variable by running the following command: 

    $ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)
    Copy to Clipboard Toggle word wrap

  4. Check the pod logs by running the following command: 

    $ oc logs -n trustee-operator-system $POD_NAME
    Copy to Clipboard Toggle word wrap

    Example output

    [2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime
    Copy to Clipboard Toggle word wrap

4.4.11. Verifying the attestation process
Copy link

You can verify the attestation process by creating a test pod and retrieving its secret.

Important

This procedure is an example to verify that attestation is working. Do not write sensitive data to standard I/O because the data can be captured by using a memory dump. Only data written to memory is encrypted.

By default, an agent side policy embedded in the pod VM image disables the exec and log APIs for a Confidential Containers pod. This policy ensures that sensitive data is not written to standard I/O.

In a test scenario, you can override the restriction at runtime by adding a policy annotation to the pod. For Technology Preview, runtime policy annotations are not verified by remote attestation.

Prerequisites

  • You have created a route if the Trustee server and the test pod are not running in the same cluster.

Procedure

  1. Create a verification-pod.yaml manifest file: 

    apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.agent.policy: cGFja2FnZSBhZ2VudF9wb2xpY3kKCmRlZmF1bHQgQWRkQVJQTmVpZ2hib3JzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgQWRkU3dhcFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENsb3NlU3RkaW5SZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBDb3B5RmlsZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZUNvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZVNhbmRib3hSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBEZXN0cm95U2FuZGJveFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEV4ZWNQcm9jZXNzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgR2V0TWV0cmljc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEdldE9PTUV2ZW50UmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgR3Vlc3REZXRhaWxzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgTGlzdEludGVyZmFjZXNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBMaXN0Um91dGVzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgTWVtSG90cGx1Z0J5UHJvYmVSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBPbmxpbmVDUFVNZW1SZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBQYXVzZUNvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFB1bGxJbWFnZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFJlYWRTdHJlYW1SZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZW1vdmVDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZW1vdmVTdGFsZVZpcnRpb2ZzU2hhcmVNb3VudHNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZXNlZWRSYW5kb21EZXZSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZXN1bWVDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBTZXRHdWVzdERhdGVUaW1lUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU2V0UG9saWN5UmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU2lnbmFsUHJvY2Vzc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFN0YXJ0Q29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU3RhcnRUcmFjaW5nUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU3RhdHNDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBTdG9wVHJhY2luZ1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFR0eVdpblJlc2l6ZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFVwZGF0ZUNvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFVwZGF0ZUVwaGVtZXJhbE1vdW50c1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFVwZGF0ZUludGVyZmFjZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFVwZGF0ZVJvdXRlc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFdhaXRQcm9jZXNzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgV3JpdGVTdHJlYW1SZXF1ZXN0IDo9IHRydWUK
    1 
    
spec:
  runtimeClassName: kata-remote
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:9.3
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
    Copy to Clipboard Toggle word wrap
    1
    This pod annotation overrides the policy that prevents sensitive data from being written to standard I/O.

  2. Create the pod by running the following command: 

    $ oc create -f verification-pod.yaml
    Copy to Clipboard Toggle word wrap

  3. Connect to the Bash shell of the ocp-cc-pod by running the following command: 

    $ oc exec -it ocp-cc-pod -- bash
    Copy to Clipboard Toggle word wrap

  4. Fetch the pod secret by running the following command: 

    $ curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1
    Copy to Clipboard Toggle word wrap

    Example output

    res1val1
    Copy to Clipboard Toggle word wrap

    The Trustee server returns the secret only if the attestation is successful.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat