Chapter 5. Deploying OpenShift sandboxed containers on Google Cloud

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

2. Create the namespace by running the following command:

   $ oc create -f osc-namespace.yaml

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

4. Create the operator group by running the following command:

   $ oc create -f osc-operatorgroup.yaml

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.10.3

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.10.3    1.9.0        Succeeded

Procedure

1. Set the project ID variable by running the following command:

   $ export GCP_PROJECT_ID="<project_id>"

2. Log in to Google Cloud by running the following command:

   $ gcloud auth login

3. Set the Google Cloud project ID by running the following command:

   $ gcloud config set project ${GCP_PROJECT_ID}

4. Open port 15150 by running the following command:

   $ gcloud compute firewall-rules create allow-port-15150-restricted \
      --project=${GCP_PROJECT_ID} \
      --network=default \
      --allow=tcp:15150 \
      --source-ranges=<external_ip_cidr-1>[,<external_ip_cidr-2>,...] 

   1

   1

   Specify one or more IP addresses or ranges in CIDR format, separated by commas. For example, 203.0.113.5/32,198.51.100.0/24.

Procedure

1. Log in to your Compute Engine instance to set the following environmental variables:

   1. Get the project ID by running the following command:

      $ GCP_PROJECT_ID=$(gcloud config get-value project)

   2. Get the zone by running the following command:

      $ GCP_ZONE=$(gcloud config get-value compute/zone)

   3. Retrieve a list of network names by running the following command:

      $ gcloud compute networks list --format="value(name)"

   4. Specify the network by running the following command:

      $ GCP_NETWORK=<network_name>

      Only auto-mode networks are supported. Custom networks are not supported at this time.

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "gcp"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     GCP_MACHINE_TYPE: "e2-medium"
     GCP_PROJECT_ID: "<project_id>"
     GCP_ZONE: "<gcp_zone>"
     GCP_NETWORK: "<gcp_network>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   GCP_MACHINE_TYPE

   Defines the default machine type that is used if the machine type is not defined in the workload object.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

Procedure

1. Clone the OpenShift sandboxed containers repository by running the following command:

   $ git clone https://github.com/openshift/sandboxed-containers-operator.git

2. Navigate to sandboxed-containers-operator/config/peerpods/podvm/bootc by running the following command:

   $ cd sandboxed-containers-operator/config/peerpods/podvm/bootc

3. Log in to registry.redhat.io by running the following command:

   $ podman login registry.redhat.io

   You must log in to registry.redhat.io, because the podman build process must access the Containerfile.rhel container image hosted on the registry.

4. Set the image path for your container registry by running the following command:

   $ IMG="<container_registry_url>/<username>/podvm-bootc:latest"

5. Build the pod VM bootc image by running the following command:

   $ podman build -t ${IMG} -f Containerfile.rhel .

6. Log in to your container registry by running the following command:

   $ podman login <container_registry_url>

7. Push the image to your container registry by running the following command:

   $ podman push ${IMG}

   For testing and development, you can make the image public.

8. Verify the podvm-bootc image by running the following command:

   $ podman images

   Example output

   REPOSITORY                               TAG     IMAGE ID      CREATED         SIZE
   example.com/example_user/podvm-bootc     latest  88ddab975a07  2 seconds ago   1.82 GB

Procedure

1. Create a podvm-image-cm.yaml manifest with the following content:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     IMAGE_TYPE: pre-built
     PODVM_IMAGE_URI: <container_registry_url>/<username>/podvm-bootc:latest
     IMAGE_BASE_NAME: "podvm-image"
     IMAGE_VERSION: "0-0-0"
   
     INSTALL_PACKAGES: "no"
     DISABLE_CLOUD_CONFIG: "true"
     UPDATE_PEERPODS_CM: "yes"
     BOOT_FIPS: "no"
   
     BOOTC_BUILD_CONFIG: |
       [[customizations.user]]
       name = "peerpod"
       password = "peerpod"
       groups = ["wheel", "root"]
   
       [[customizations.filesystem]]
       mountpoint = "/"
       minsize = "5 GiB"
   
       [[customizations.filesystem]]
       mountpoint = "/var/kata-containers"
       minsize = "15 GiB"

2. Create the config map by running the following command:

   $ oc create -f podvm-image-cm.yaml

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default ExecProcessRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SetPolicyRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default WriteStreamRequest := true

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>' 

   1

   1

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, osc: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Example output

   NAME             HANDLER          AGE
   kata-remote      kata-remote      152m

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

2. Check the status stanza of the YAML file.

   If the PODVM_IMAGE_NAME parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   OpenShift Container Platform creates the workload object and begins scheduling it.