Chapter 6. Deploying OpenShift sandboxed containers on IBM Z and IBM LinuxONE

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

   Copy to Clipboard Toggle word wrap

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

   Copy to Clipboard Toggle word wrap

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the KVM host.

2. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   Copy to Clipboard Toggle word wrap

   You need the LIBVIRT_POOL value to create the secret for the libvirt provider.

3. Set the name of the libvirt volume by running the following command:

   $ export LIBVIRT_VOL_NAME=<libvirt_volume>

   Copy to Clipboard Toggle word wrap

   You need the LIBVIRT_VOL_NAME value to create the secret for the libvirt provider.

4. Set the path of the default storage pool location, by running the following command:

   $ export LIBVIRT_POOL_DIRECTORY="/var/lib/libvirt/images/"

   Copy to Clipboard Toggle word wrap

5. Create a libvirt pool by running the following command:

   $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

   Copy to Clipboard Toggle word wrap

6. Start the libvirt pool by running the following command:

   $ virsh pool-start $LIBVIRT_POOL

   Copy to Clipboard Toggle word wrap

7. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Dockerfile.podvm-oci file:

   FROM scratch
   
   ARG PODVM_IMAGE_SRC
   ENV PODVM_IMAGE_PATH="/image/podvm.qcow2"
   
   COPY $PODVM_IMAGE_SRC $PODVM_IMAGE_PATH

   Copy to Clipboard Toggle word wrap

2. Build a container with the pod VM QCOW2 image by running the following command:

   $ docker build -t podvm-libvirt \
     --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 

   1

   
     --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 

   2

   
     -f Dockerfile.podvm-oci .

   Copy to Clipboard Toggle word wrap

   1

   Specify the QCOW2 image source on the host.

   2

   Optional: Specify the path of the QCOW2 image if you do not use the default, /image/podvm.qcow2.

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_URI: "<libvirt_gateway_uri>" 

   1

   
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the libvirt URI.

   2

   Specify the Red Hat offline token, which is required for the Operator-built image.

2. Create the secret by running the following command:

   $ oc create -f peer-pods-secret.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_POOL: "<libvirt_pool>"
     LIBVIRT_VOL_NAME: "<libvirt_volume>"
     LIBVIRT_DIR_NAME: "/var/lib/libvirt/images/<directory_name>"
     LIBVIRT_NET: "default"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy to Clipboard Toggle word wrap

   LIBVIRT_POOL

   If you have manually configured the libvirt pool, use the same name as in your KVM host configuration.

   LIBVIRT_VOL_NAME

   If you have manually configured the libvirt volume, use the same name as in your KVM host configuration.

   LIBVIRT_DIR_NAME

   Specify the libvirt directory for storing virtual machine disk images, such as .qcow2, or .raw files. To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.

   LIBVIRT_NET

   Specify a libvirt network if you do not want to use the default network.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

2. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a libvirt-podvm-image-cm.yaml manifest according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: libvirt-podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     PODVM_DISTRO: "rhel"
     DOWNLOAD_SOURCES: "no" 

   1

   
     CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor" 

   2

   
     CAA_REF: "main" 

   3

   
     CONFIDENTIAL_COMPUTE_ENABLED: "yes"
     UPDATE_PEERPODS_CM: "yes"
     ORG_ID: "<rhel_organization_id>"
     ACTIVATION_KEY: "<rhel_activation_key>" 

   4

   
     PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 

   5

   
     SE_BOOT: "true" 

   6

   
     BASE_OS_VERSION: "<rhel_image_os_version>" 

   7

   
     SE_VERIFY: "false" 

   8

   Copy to Clipboard Toggle word wrap

   1

   Specify yes if you want to use the custom Cloud API Adaptor source to build the pod VM image.

   2

   Optional: Specify the URL of the Cloud API Adaptor custom image.

   3

   Optional: Specify the branch or tag of the Cloud API Adaptor custom image.

   4

   Specify your RHEL activation key.

   5

   Optional: If you created a custom peer pod VM image, specify the container registry URL, the image tag, and the image path (default: /image/podvm.qcow2). Otherwise, set the value to "".

   6

   The default value, true, enables IBM Secure Execution for the default Operator-built image. If you use a custom peer pod VM image, set it to false .

   7

   Specify the RHEL image operating system version. IBM Z® Secure Execution supports RHEL 9.5 and later versions.

   8

   Specify false if you do not want to verify Secure Execution with the digicert CA certificate. The default value is true.

2. Create the config map by running the following command:

   $ oc create -f libvirt-podvm-image-cm.yaml

   Copy to Clipboard Toggle word wrap

   The libvirt pod VM image config map is created for your libvirt provider.

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

   Copy to Clipboard Toggle word wrap

2. Copy the public SSH key to your KVM host:

   $ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_IP> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the IP address of your KVM host or the LPAR where the peer pod VM is running. For example, 192.168.122.1.

3. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

   Copy to Clipboard Toggle word wrap

4. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the LIBVIRT_POOL variable by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   Copy to Clipboard Toggle word wrap

2. Set the LIBVIRT_VOL_NAME variable to a new libvirt volume by running the following command:

   $ export LIBVIRT_VOL_NAME=<new_libvirt_volume>

   Copy to Clipboard Toggle word wrap

3. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

   Copy to Clipboard Toggle word wrap

4. Upload the custom peer pod VM image to the new libvirt volume:

   $ virsh -c qemu:///system vol-upload \
     --vol $LIBVIRT_VOL_NAME <custom_podvm_image.qcow2> \
     --pool $LIBVIRT_POOL --sparse

   Copy to Clipboard Toggle word wrap

5. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<new_libvirt_volume>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy to Clipboard Toggle word wrap

6. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true

   Copy to Clipboard Toggle word wrap

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy to Clipboard Toggle word wrap

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy to Clipboard Toggle word wrap

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy to Clipboard Toggle word wrap

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Copy to Clipboard Toggle word wrap

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   
   BinaryData
   ====
   Events: <none>

   Copy to Clipboard Toggle word wrap

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

   Copy to Clipboard Toggle word wrap

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy to Clipboard Toggle word wrap

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy to Clipboard Toggle word wrap

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy to Clipboard Toggle word wrap

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy to Clipboard Toggle word wrap

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy to Clipboard Toggle word wrap

   OpenShift Container Platform creates the workload object and begins scheduling it.