Chapter 2. Deploying confidential containers on Azure


You deploy confidential containers on a Red Hat OpenShift Container Platform cluster on Microsoft Azure Cloud Computing Services for your workloads.

You deploy confidential containers by performing the following steps:

  1. Configure outbound connections.
  2. Install the OpenShift sandboxed containers Operator.
  3. Enable the confidential containers feature gate.
  4. Optional: If you pull a peer pod VM image from a private registry such as registry.access.redhat.com, configure the pull secret for peer pods.
  5. Create initdata. See About initdata for details.
  6. Create the peer pods config map.
  7. Optional: Apply initdata to a peer pod.
  8. Create the KataConfig CR.
  9. Verify the attestation process.

2.1. Prerequisites

  • You have installed Red Hat OpenShift Container Platform 4.16 or later on the cluster where you are running your confidential containers workload.
  • You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.
  • You have enabled ports 15150 and 9000 for communication in the subnet used for worker nodes and the pod virtual machine (VM). The ports enable communication between the Kata shim running on the worker node and the Kata agent running on the pod VM.
  • You have configured outbound connectivity for the pod VM subnet.

2.2. Configuring outbound connections

To enable peer pods to communicate with external networks, such as the public internet, you must configure outbound connectivity for the pod virtual machine (VM) subnet. This involves setting up a NAT gateway and, optionally, defining how the subnet integrates with your cluster’s virtual network (VNet) in Azure.

Peer pods and subnets
Peer pods operate in a dedicated Azure subnet that requires explicit configuration for outbound access. This subnet can either be the default worker subnet used by OpenShift Container Platform nodes or a separate, custom subnet created specifically for peer pods.
VNet peering
When using a separate subnet, VNet peering connects the peer pod VNet to the cluster’s VNet, ensuring internal communication while maintaining isolation. This requires non-overlapping CIDR ranges between the VNets.

You can configure outbound connectivity in two ways:

  • Default worker subnet: Modify the existing worker subnet to include a NAT gateway. This is simpler and reuses cluster resources, but it offers less isolation.
  • Peer pod VNet: Set up a dedicated VNet and subnet for peer pods, attach a NAT gateway, and peer it with the cluster VNet. This provides greater isolation and flexibility at the cost of additional complexity.

You can configure the default worker subnet with a NAT gateway.

Prerequisites

  • The Azure CLI (az) is installed and authenticated.
  • You have administrator access to the Azure resource group and the VNet.

Procedure

  1. Set the AZURE_RESOURCE_GROUP environment variable by running the following command:

    $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')
    Copy to Clipboard Toggle word wrap
  2. Set the AZURE_REGION environment variable by running the following command:

    $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
        --query "{Location:location}" --output tsv) && \
        echo "AZURE_REGION: \"$AZURE_REGION\""
    Copy to Clipboard Toggle word wrap
  3. Set the AZURE_VNET_NAME environment variable by running the following command:

    $ AZURE_VNET_NAME=$(az network vnet list \
        -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)
    Copy to Clipboard Toggle word wrap
  4. Set the AZURE_SUBNET_ID environment variable by running the following command:

    $ AZURE_SUBNET_ID=$(az network vnet subnet list \
        --resource-group "${AZURE_RESOURCE_GROUP}" \
        --vnet-name "${AZURE_VNET_NAME}" --query "[].{Id:id} \
        | [? contains(Id, 'worker')]" --output tsv)
    Copy to Clipboard Toggle word wrap
  5. Set the NAT gateway environment variables for the peer pod subnet by running the following commands:

    $ export PEERPOD_NAT_GW=peerpod-nat-gw
    Copy to Clipboard Toggle word wrap
    $ export PEERPOD_NAT_GW_IP=peerpod-nat-gw-ip
    Copy to Clipboard Toggle word wrap
  6. Create a public IP address for the NAT gateway by running the following command:

    $ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
        -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}" --sku Standard
    Copy to Clipboard Toggle word wrap
  7. Create the NAT gateway and associate it with the public IP address by running the following command:

    $ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
        -l "${AZURE_REGION}" --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
        -n "${PEERPOD_NAT_GW}"
    Copy to Clipboard Toggle word wrap
  8. Update the VNet subnet to use the NAT gateway by running the following command:

    $ az network vnet subnet update --nat-gateway "${PEERPOD_NAT_GW}" \
        --ids "${AZURE_SUBNET_ID}"
    Copy to Clipboard Toggle word wrap

Verification

  • Confirm the NAT gateway is attached to the VNet subnet by running the following command:

    $ az network vnet subnet show --ids "${AZURE_SUBNET_ID}" \
        --query "natGateway.id" -o tsv
    Copy to Clipboard Toggle word wrap

    The output contains the NAT gateway resource ID. If no NAT gateway is attached, the output is empty.

    Example output

    /subscriptions/12345678-1234-1234-1234-1234567890ab/resourceGroups/myResourceGroup/providers/Microsoft.Network/natGateways/myNatGateway
    Copy to Clipboard Toggle word wrap

To enable public internet access, you can create a dedicated virtual network (VNet) for peer pods, attach a network address translation (NAT) gateway, create a subnet, and enable VNet peering with non-overlapping address spaces.

Prerequisites

  • The Azure CLI (az) is installed
  • You have signed in to Azure. See Authenticate to Azure using Azure CLI.
  • You have administrator access to the Azure resource group and VNet hosting the cluster.
  • You have verified the cluster VNet classless inter-domain routing (CIDR) address. The default value is 10.0.0.0/14. If you overrode the default value, you have ensured that you chose a non-overlapping CIDR address for the peer pod VNet. For example, 192.168.0.0/16.

Procedure

  1. Set the environmental variables for the peer pod network:

    1. Set the peer pod VNet environment variables by running the following commands:

      $ export PEERPOD_VNET_NAME="${PEERPOD_VNET_NAME:-peerpod-vnet}"
      Copy to Clipboard Toggle word wrap
      $ export PEERPOD_VNET_CIDR="${PEERPOD_VNET_CIDR:-192.168.0.0/16}"
      Copy to Clipboard Toggle word wrap
    2. Set the peer pod subnet environment variables by running the following commands:

      $ export PEERPOD_SUBNET_NAME="${PEERPOD_SUBNET_NAME:-peerpod-subnet}"
      Copy to Clipboard Toggle word wrap
      $ export PEERPOD_SUBNET_CIDR="${PEERPOD_SUBNET_CIDR:-192.168.0.0/16}"
      Copy to Clipboard Toggle word wrap
  2. Set the environmental variables for Azure:

    $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')
    Copy to Clipboard Toggle word wrap
    $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
        --query "{Location:location}" --output tsv) && \
        echo "AZURE_REGION: \"$AZURE_REGION\""
    Copy to Clipboard Toggle word wrap
    $ AZURE_VNET_NAME=$(az network vnet list \
        -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)
    Copy to Clipboard Toggle word wrap
  3. Set the peer pod NAT gateway environment variables by running the following commands:

    $ export PEERPOD_NAT_GW="${PEERPOD_NAT_GW:-peerpod-nat-gw}"
    Copy to Clipboard Toggle word wrap
    $ export PEERPOD_NAT_GW_IP="${PEERPOD_NAT_PUBLIC_IP:-peerpod-nat-gw-ip}"
    Copy to Clipboard Toggle word wrap
  4. Configure the VNET:

    1. Create the peer pod VNet by running the following command:

      $ az network vnet create --resource-group "${AZURE_RESOURCE_GROUP}" \
          --name "${PEERPOD_VNET_NAME}" \
          --address-prefixes "${PEERPOD_VNET_CIDR}"
      Copy to Clipboard Toggle word wrap
    2. Create a public IP address for the peer pod VNet by running the following command:

      $ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
          -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}"
      Copy to Clipboard Toggle word wrap
    3. Create a NAT gateway for the peer pod VNet by running the following command:

      $ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
          -l "${AZURE_REGION}" \
          --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
          -n "${PEERPOD_NAT_GW}"
      Copy to Clipboard Toggle word wrap
    4. Create a subnet in the peer pod VNet and attach the NAT gateway by running the following command:

      $ az network vnet subnet create \
          --resource-group "${AZURE_RESOURCE_GROUP}" \
          --vnet-name "${PEERPOD_VNET_NAME}" \
          --name "${PEERPOD_SUBNET_NAME}" \
          --address-prefixes "${PEERPOD_SUBNET_CIDR}" \
          --nat-gateway "${PEERPOD_NAT_GW}"
      Copy to Clipboard Toggle word wrap
  5. Configure the virtual network peering connection:

    1. Create the peering connection by running the following command:

      $ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-azure-vnet-to-peerpod-vnet \
          --vnet-name "${AZURE_VNET_NAME}" \
          --remote-vnet "${PEERPOD_VNET_NAME}" --allow-vnet-access \
          --allow-forwarded-traffic
      Copy to Clipboard Toggle word wrap
    2. Sync the peering connection by running the following command:

      $ az network vnet peering sync -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-azure-vnet-to-peerpod-vnet \
          --vnet-name "${AZURE_VNET_NAME}"
      Copy to Clipboard Toggle word wrap
    3. Complete the peering connection by running the following command:

      $ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
          -n peerpod-peerpod-vnet-to-azure-vnet \
          --vnet-name "${PEERPOD_VNET_NAME}" \
          --remote-vnet "${AZURE_VNET_NAME}" --allow-vnet-access \
          --allow-forwarded-traffic
      Copy to Clipboard Toggle word wrap

Verification

  1. Check the peering connection status from the cluster VNet by running the following command:

    $ az network vnet peering show -g "${AZURE_RESOURCE_GROUP}" \
        -n peerpod-azure-vnet-to-peerpod-vnet \
        --vnet-name "${AZURE_VNET_NAME}" \
        --query "peeringState" -o tsv
    Copy to Clipboard Toggle word wrap

    This should return Connected.

  2. Verify that the NAT gateway is attached to the peer pod subnet by running the following command:

    $ az network vnet subnet show --resource-group "${AZURE_RESOURCE_GROUP}" \
        --vnet-name "${PEERPOD_VNET_NAME}" --name "${PEERPOD_SUBNET_NAME}" \
        --query "natGateway.id" -o tsv
    Copy to Clipboard Toggle word wrap

You install the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create an osc-namespace.yaml manifest file:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap
  2. Create the namespace by running the following command:

    $ oc apply -f osc-namespace.yaml
    Copy to Clipboard Toggle word wrap
  3. Create an osc-operatorgroup.yaml manifest file:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: sandboxed-containers-operator-group
      namespace: openshift-sandboxed-containers-operator
    spec:
      targetNamespaces:
      - openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap
  4. Create the operator group by running the following command:

    $ oc apply -f osc-operatorgroup.yaml
    Copy to Clipboard Toggle word wrap
  5. Create an osc-subscription.yaml manifest file:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: sandboxed-containers-operator
      namespace: openshift-sandboxed-containers-operator
    spec:
      channel: stable
      installPlanApproval: Automatic
      name: sandboxed-containers-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      startingCSV: sandboxed-containers-operator.v1.10.1
    Copy to Clipboard Toggle word wrap
  6. Create the subscription by running the following command:

    $ oc apply -f osc-subscription.yaml
    Copy to Clipboard Toggle word wrap
  7. Verify that the Operator is correctly installed by running the following command:

    $ oc get csv -n openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

    This command can take several minutes to complete.

  8. Watch the process by running the following command:

    $ watch oc get csv -n openshift-sandboxed-containers-operator
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.1    1.9.0        Succeeded
    Copy to Clipboard Toggle word wrap

You enable the confidential containers feature gate by creating the osc-feature-gates config map.

Procedure

  1. Create a cc-feature-gate.yaml manifest file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: osc-feature-gates
      namespace: openshift-sandboxed-containers-operator
    data:
      confidential: "true"
    Copy to Clipboard Toggle word wrap
  2. Create the osc-feature-gates config map by running the following command:

    $ oc create -f cc-feature-gate.yaml
    Copy to Clipboard Toggle word wrap

2.5. Configuring the pull secret for peer pods

To pull pod VM images from a private registry, you must configure the pull secret for peer pods.

Then, you can link the pull secret to the default service account or you can specify the pull secret in the peer pod manifest.

Procedure

  1. Set the NS variable to the namespace where you deploy your peer pods:

    $ NS=<namespace>
    Copy to Clipboard Toggle word wrap
  2. Copy the pull secret to the peer pod namespace:

    $ oc get secret pull-secret -n openshift-config -o yaml \
      | sed "s/namespace: openshift-config/namespace: ${NS}/" \
      | oc apply -n "${NS}" -f -
    Copy to Clipboard Toggle word wrap

    You can use the cluster pull secret, as in this example, or a custom pull secret.

  3. Optional: Link the pull secret to the default service account:

    $ oc secrets link default pull-secret --for=pull -n ${NS}
    Copy to Clipboard Toggle word wrap
  4. Alternatively, add the pull secret to the peer pod manifest:

    apiVersion: v1
    kind: <Pod>
    spec:
      containers:
      - name: <container_name>
        image: <image_name>
      imagePullSecrets:
      - name: pull-secret
    # ...
    Copy to Clipboard Toggle word wrap

2.6. Creating initdata

You create an initdata.toml file and convert it to a Base64-encoded string in gzip format.

You specify this string as the INITDATA value in the peer pods config map, for global configuration, or as an annotation in a peer pod manifest, for a specific pod.

Then, you generate a Platform Configuration Register (PCR) 8 hash from the initdata.toml file for the Reference Value Provider Service (RVPS) config map for Red Hat build of Trustee.

Red Hat build of Trustee uses the RVPS to validate attestation evidence sent by confidential workloads. The RVPS contains trusted reference values, such as file hashes, that are compared to the PCR measurements included in attestation requests. These hashes are not generated by Red Hat build of Trustee.

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map.

Procedure

  1. Obtain the Red Hat build of Trustee URL by running the following command:

    $ TRUSTEE_URL=$(oc get route kbs-service \
      -n trustee-operator-system -o jsonpath='{.spec.host}') \
      && echo $TRUSTEE_URL
    Copy to Clipboard Toggle word wrap
  2. Create the initdata.toml file:

    algorithm = "sha384"
    version = "0.1.0"
    
    [data]
    "aa.toml" = '''
    [token_configs]
    [token_configs.coco_as]
    
    url = '<trustee_url>'
    
    [token_configs.kbs]
    url = '<trustee_url>'
    cert = """
    -----BEGIN CERTIFICATE-----
    <kbs_certificate>
    -----END CERTIFICATE-----
    """
    '''
    
    "cdh.toml" = '''
    socket = 'unix:///run/confidential-containers/cdh.sock'
    credentials = []
    
    [kbc]
    name = 'cc_kbc'
    url = '<trustee_url>'
    kbs_cert = """
    -----BEGIN CERTIFICATE-----
    <kbs_certificate>
    -----END CERTIFICATE-----
    """
    '''
    
    "policy.rego" = '''
    package agent_policy
    
    default AddARPNeighborsRequest := true
    default AddSwapRequest := true
    default CloseStdinRequest := true
    default CopyFileRequest := true
    default CreateContainerRequest := true
    default CreateSandboxRequest := true
    default DestroySandboxRequest := true
    default ExecProcessRequest := false
    default GetMetricsRequest := true
    default GetOOMEventRequest := true
    default GuestDetailsRequest := true
    default ListInterfacesRequest := true
    default ListRoutesRequest := true
    default MemHotplugByProbeRequest := true
    default OnlineCPUMemRequest := true
    default PauseContainerRequest := true
    default PullImageRequest := true
    default ReadStreamRequest := false
    default RemoveContainerRequest := true
    default RemoveStaleVirtiofsShareMountsRequest := true
    default ReseedRandomDevRequest := true
    default ResumeContainerRequest := true
    default SetGuestDateTimeRequest := true
    default SetPolicyRequest := true
    default SignalProcessRequest := true
    default StartContainerRequest := true
    default StartTracingRequest := true
    default StatsContainerRequest := true
    default StopTracingRequest := true
    default TtyWinResizeRequest := true
    default UpdateContainerRequest := true
    default UpdateEphemeralMountsRequest := true
    default UpdateInterfaceRequest := true
    default UpdateRoutesRequest := true
    default WaitProcessRequest := true
    default WriteStreamRequest := true
    '''
    Copy to Clipboard Toggle word wrap
    • <trustee-url>: Specify the Red Hat build of Trustee URL. If you configure the Red Hat build of Trustee with insecure_http for testing purposes, use HTTP. Otherwise, use HTTPS. For production systems, avoid using insecure_http unless you configure your environment to handle TLS externally, for example, with a proxy.
    • <kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
    • kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map.
  3. Convert the initdata.toml file to a Base64-encoded string in gzip format in a text file by running the following command:

    $ cat initdata.toml | gzip | base64 -w0 > initdata.txt
    Copy to Clipboard Toggle word wrap

    Record this string for the peer pods config map or a peer pod manifest.

  4. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

    $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)
    Copy to Clipboard Toggle word wrap
  5. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

    $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
    Copy to Clipboard Toggle word wrap
  6. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

    $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
    Copy to Clipboard Toggle word wrap

    Record the PCR8_HASH value for the. Calculate the SHA-256 hash of an initdata.toml file and assign its value to the hash variable by running the following command:

    $ hash=$(sha256sum initdata.toml | cut -d' ' -f1)
    Copy to Clipboard Toggle word wrap
  7. Assign 32 bytes of 0s to the initial_pcr variable by running the following command:

    $ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
    Copy to Clipboard Toggle word wrap
  8. Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:

    $ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
    Copy to Clipboard Toggle word wrap

    Record the PCR8_HASH value for the RVPS config map.

2.7. Creating the peer pods config map

You must create the peer pods config map.

Procedure

  1. Obtain the following values from your Azure instance:

    1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
        && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
      Copy to Clipboard Toggle word wrap
    2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Name:name}" --output tsv)
      Copy to Clipboard Toggle word wrap

      This value is used to retrieve the Azure subnet ID.

    3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
        --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
         && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
      Copy to Clipboard Toggle word wrap
    4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
      Copy to Clipboard Toggle word wrap
    5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "{Location:location}" --output tsv) \
        && echo "AZURE_REGION: \"$AZURE_REGION\""
      Copy to Clipboard Toggle word wrap
  2. Create a peer-pods-cm.yaml manifest file according to the following example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "azure"
      VXLAN_PORT: "9000"
      PROXY_TIMEOUT: "5m"
      AZURE_INSTANCE_SIZE: "Standard_DC2as_v5"
      AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5"
      AZURE_SUBNET_ID: "<azure_subnet_id>"
      AZURE_NSG_ID: "<azure_nsg_id>"
      AZURE_IMAGE_ID: ""
      AZURE_REGION: "<azure_region>"
      AZURE_RESOURCE_GROUP: "<azure_resource_group>"
      TAGS: "key1=value1,key2=value2"
      PEERPODS_LIMIT_PER_NODE: "10"
      ROOT_VOLUME_SIZE: "6"
      DISABLECVM: "false"
      INITDATA: "<initdata_string>"
    Copy to Clipboard Toggle word wrap
    AZURE_INSTANCE_SIZE
    Defines the default instance size that is used if the instance size is not defined in the workload object. "Standard_DC2as_v5" is for AMD SEV-SNP. If your TEE is Intel TDX, specify Standard_EC4eds_v5.
    AZURE_IMAGE_ID
    Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod VM image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster. This image provides root disk integrity protection (dm-verity) and encrypted container storage. See Confidential VMs: The core of confidential containers for details.
    AZURE_INSTANCE_SIZES
    Specify the instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
    TAGS
    You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.
    PEERPODS_LIMIT_PER_NODE
    You can increase this value to run more peer pods on a node. The default value is 10.
    ROOT_VOLUME_SIZE
    You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.
  3. Create the config map by running the following command:

    $ oc create -f peer-pods-cm.yaml
    Copy to Clipboard Toggle word wrap

2.8. Applying initdata to a pod

You can override the global INITDATA setting you applied in the peer pods config map by applying customized initdata to a specific pod for special use cases, such as development and testing with a relaxed policy, or when using different Red Hat build of Trustee configurations. You can customize initdata by adding an annotation to the workload pod YAML.

Prerequisite

  • You have created an initdata string.

Procedure

  1. Add the initdata string to the pod manifest:

    apiVersion: v1
    kind: Pod
    metadata:
      name: ocp-cc-pod
      labels:
        app: ocp-cc-pod
      annotations:
        io.katacontainers.config.runtime.cc_init_data: <initdata_string>
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: <container_name>
        image: registry.access.redhat.com/ubi9/ubi:latest
        command:
        - sleep
        - "36000"
        securityContext:
          privileged: false
          seccompProfile:
            type: RuntimeDefault
    Copy to Clipboard Toggle word wrap
  2. Apply the pod manifest by running the following command:

    $ oc apply -f my-pod.yaml
    Copy to Clipboard Toggle word wrap

2.9. Selecting a custom peer pod VM image

You can select a custom peer pod virtual machine (VM) image, tailored to your workload requirements by adding an annotation to the pod manifest. The custom image overrides the default image specified in the peer pods config map.

Prerequisites

  • You have the ID of a custom pod VM image, which is compatible with your cloud provider or hypervisor.

Procedure

  1. Create a my-pod-manifest.yaml file according to the following example:

    apiVersion: v1
    kind: Pod
    metadata:
      name: my-pod-manifest
      annotations:
        io.katacontainers.config.hypervisor.image: "<custom_image_id>"
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: <example_container>
        image: registry.access.redhat.com/ubi9/ubi:9.3
        command: ["sleep", "36000"]
    Copy to Clipboard Toggle word wrap
  2. Create the pod by running the following command:

    $ oc create -f my-pod-manifest.yaml
    Copy to Clipboard Toggle word wrap

2.10. Creating the KataConfig custom resource

You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

  • A large OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Procedure

  1. Create an example-kataconfig.yaml manifest file according to the following example:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: example-kataconfig
    spec:
      enablePeerPods: true
      logLevel: info
    #  kataConfigPoolSelector:
    #    matchLabels:
    #      <label_key>: '<label_value>' 
    1
    Copy to Clipboard Toggle word wrap
    1
    Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, cc: 'true'.
  2. Create the KataConfig CR by running the following command:

    $ oc apply -f example-kataconfig.yaml
    Copy to Clipboard Toggle word wrap

    The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

    Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

  3. Monitor the installation progress by running the following command:

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
    Copy to Clipboard Toggle word wrap

    When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

  4. Verify the daemon set by running the following command:

    $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds
    Copy to Clipboard Toggle word wrap
  5. Verify the runtime classes by running the following command:

    $ oc get runtimeclass
    Copy to Clipboard Toggle word wrap

    Example output

    NAME             HANDLER          AGE
    kata-remote      kata-remote      152m
    Copy to Clipboard Toggle word wrap

2.11. Verifying attestation

You can verify the attestation process by creating a test pod with a relaxed Kata agent policy and retrieving its key.

Important

This procedure is an example to verify that attestation is working. Do not write sensitive data to standard I/O, because the data can be captured by using a memory dump. Only data written to memory is encrypted.

Procedure

  1. Create a test-pod.yaml manifest file:

    apiVersion: v1
    kind: Pod
    metadata:
      name: ocp-cc-pod
      labels:
        app: ocp-cc-pod
      annotations:
        io.katacontainers.config.runtime.cc_init_data: <initdata_string> 
    1
    
    spec:
      runtimeClassName: kata-remote
      containers:
        - name: skr-openshift
          image: registry.access.redhat.com/ubi9/ubi:9.3
          command:
            - sleep
            - "36000"
          securityContext:
            privileged: false
            seccompProfile:
              type: RuntimeDefault
    Copy to Clipboard Toggle word wrap
    1
    Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.
  2. Create the pod by running the following command:

    $ oc create -f test-pod.yaml
    Copy to Clipboard Toggle word wrap
  3. Log in to the pod by running the following command:

    $ oc exec -it ocp-cc-pod -- bash
    Copy to Clipboard Toggle word wrap
  4. Fetch the pod secret by running the following command:

    $ curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1
    Copy to Clipboard Toggle word wrap

    Example output

    res1val1/ #
    Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat