OpenShift sandboxed containers user guide

Procedure

1. Create a NodeFeatureDiscovery resource to detect node capabilities suitable for running Kata containers:

   1. Save the following YAML in the nfd.yaml file:

      apiVersion: nfd.openshift.io/v1
      kind: NodeFeatureDiscovery
      metadata:
        name: nfd-kata
        namespace: openshift-nfd
      spec:
        operand:
          image: quay.io/openshift/origin-node-feature-discovery:4.10
          imagePullPolicy: Always
          servicePort: 12000
        workerConfig:
          configData: |
            sources:
               custom:
                 - name: "feature.node.kubernetes.io/runtime.kata"
                   matchOn:
                     - cpuId: ["SSE4", "VMX"]
                       loadedKMod: ["kvm", "kvm_intel"]
                     - cpuId: ["SSE4", "SVM"]
                       loadedKMod: ["kvm", "kvm_amd"]

   2. Create the NodeFeatureDiscovery custom resource (CR):

      $ oc create -f nfd.yaml

      Example output

      nodefeaturediscovery.nfd.openshift.io/nfd-kata created

      A feature.node.kubernetes.io/runtime.kata=true label is applied to all qualifying worker nodes.

2. Set the checkNodeEligibility field to true in the KataConfig resource to enable the feature, for example:

   1. Save the following YAML in the kata-config.yaml file:

      apiVersion: kataconfiguration.openshift.io/v1
      kind: KataConfig
      metadata:
        name: example-kataconfig
      spec:
        checkNodeEligibility: true

   2. Create the KataConfig CR:

      $ oc create -f kata-config.yaml

      Example output

      kataconfig.kataconfiguration.openshift.io/example-kataconfig created

Procedure

1. From the Administrator perspective in the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers tile.
4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Select stable from the list of available Update Channel options.

   2. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

      Note

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

   3. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
6. Click Install.

Verification

1. From the Administrator perspective in the web console, navigate to Operators → Installed Operators.
2. Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. In the KataConfig tab, click Create KataConfig.

4. In the Create KataConfig page, enter the following details:

   • Name: Enter a name for the KataConfig resource. By default, the name is defined as example-kataconfig.
   • Labels (Optional): Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • checkNodeEligibility (Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
   • enablePeerPods (For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment.

   • kataConfigPoolSelector: By default, kata is installed as a RuntimeClass on all nodes. If you want to install kata as a RuntimeClass only on selected nodes, you must add a matchExpression:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the key field, add the label key the selector applies to.
     5. In the operator field, add the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the values area, and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes running kata as a RuntimeClass. For more information, see "Collecting OpenShift sandboxed containers data".
5. Click Create.

Verification

1. In the KataConfig tab, select the new KataConfig CR.
2. In the KataConfig page, select the YAML tab.

3. Monitor the status field.

   A message appears each time there is an update. Click Reload to view the updated KataConfig CR.

   The status field has conditions and kataNodes subfields. The kataNodes subfield is a collection of node lists, each of which lists nodes in a particular state of kata installation.

   When all workers under kataNodes are listed as installed, and the condition InProgress is False without specifying a reason, this indicates that kata is installed on the cluster. For more information, see "Installation and uninstall transitions".

Procedure

1. From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
2. In the workload page, click to create the workload.

3. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata.

   Example for Pod object

   apiVersion: v1
   kind: Pod
   metadata:
     name: hello-openshift
     labels:
       app: hello-openshift
   spec:
     runtimeClassName: kata
     containers:
       - name: hello-openshift
         image: quay.io/openshift/origin-hello-openshift
         ports:
           - containerPort: 8888
         securityContext:
           privileged: false
           allowPrivilegeEscalation: false
           runAsNonRoot: true
           runAsUser: 1001
           capabilities:
             drop:
               - ALL
           seccompProfile:
             type: RuntimeDefault

4. Click Save.

Procedure

1. Create the Namespace object for the OpenShift sandboxed containers Operator.

   1. Create a Namespace object YAML file that contains the following manifest:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-sandboxed-containers-operator

   2. Create the Namespace object:

      $ oc create -f Namespace.yaml

2. Create the OperatorGroup object for the OpenShift sandboxed containers Operator.

   1. Create an OperatorGroup object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        targetNamespaces:
        - openshift-sandboxed-containers-operator

   2. Create the OperatorGroup object:

      $ oc create -f OperatorGroup.yaml

3. Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator.

   1. Create a Subscription object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: sandboxed-containers-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: sandboxed-containers-operator.v1.5.3

   2. Create the Subscription object:

      $ oc create -f Subscription.yaml

Procedure

1. Create a YAML file with the following manifest:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: cluster-kataconfig
   spec:
     checkNodeEligibility: false 1
     logLevel: info

   1

   Set`checkNodeEligibility` to true to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".

2. (Optional) If you want to install kata as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: cluster-kataconfig
   spec:
     checkNodeEligibility: false
     logLevel: info
     kataConfigPoolSelector:
       matchLabels:
         <label_key>: '<label_value>' 1

   1

   Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported.

3. Create the KataConfig resource:

   $ oc create -f cluster-kataconfig.yaml

Procedure

1. From the Administrator perspective in the web console, navigate to Operators → OperatorHub.
2. In the Filter by keyword field, type OpenShift sandboxed containers.
3. Select the OpenShift sandboxed containers tile.
4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Select stable from the list of available Update Channel options.

   2. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

      Note

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

   3. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
6. Click Install.

Verification

1. From the Administrator perspective in the web console, navigate to Operators → Installed Operators.
2. Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.

Procedure

1. From the Administrator perspective in the web console, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator from the list of operators.
3. In the KataConfig tab, click Create KataConfig.

4. In the Create KataConfig page, enter the following details:

   • Name: Enter a name for the KataConfig resource. By default, the name is defined as example-kataconfig.
   • Labels (Optional): Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
   • checkNodeEligibility (Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
   • enablePeerPods (For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment.

   • kataConfigPoolSelector: By default, kata-remote is installed as a RuntimeClass on all nodes. If you want to install kata-remote as a RuntimeClass only on selected nodes, you must add a matchExpression:

     1. Expand the kataConfigPoolSelector area.
     2. In the kataConfigPoolSelector, expand matchExpressions. This is a list of label selector requirements.
     3. Click Add matchExpressions.
     4. In the key field, add the label key the selector applies to.
     5. In the operator field, add the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
     6. Expand the values area, and then click Add value.
     7. In the Value field, enter true or false for key label value.
   • logLevel: Define the level of log data retrieved for nodes running kata-remote as a RuntimeClass. For more information, see "Collecting OpenShift sandboxed containers data".
5. Click Create.

Verification

1. In the KataConfig tab, select the new KataConfig CR.
2. In the KataConfig page, select the YAML tab.

3. Monitor the status field.

   A message appears each time there is an update. Click Reload to view the updated KataConfig CR.

   The status field has conditions and kataNodes subfields. The kataNodes subfield is a collection of node lists, each of which lists nodes in a particular state of kata installation.

   When all workers under kataNodes are listed as installed, and the condition InProgress is False without specifying a reason, this indicates that kata is installed on the cluster. For more information, see "Installation and uninstall transitions".

Procedure

1. From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
2. In the workload page, click to create the workload.
3. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata-remote.

4. In the YAML file for the workload, add an annotation to define whether to use a default instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.

   • For a specific instance size type, add the following annotation:

     io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>

     Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the ConfigMap for peer pods. Choose from one of those.

   • For an automatic instance size or type, add the following annotations:

     io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
     io.katacontainers.config.hypervisor.default_memory: <memory>

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.

   Example for Pod object

   apiVersion: v1
   kind: Pod
   metadata:
     name: hello-openshift
     labels:
       app: hello-openshift
     annotations:
       io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1
   spec:
     runtimeClassName: kata-remote
     containers:
       - name: hello-openshift
         image: quay.io/openshift/origin-hello-openshift
         ports:
           - containerPort: 8888
         securityContext:
           privileged: false
           allowPrivilegeEscalation: false
           runAsNonRoot: true
           runAsUser: 1001
           capabilities:
             drop:
               - ALL
           seccompProfile:
             type: RuntimeDefault

   1

   This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.

5. Click Save.

Procedure

1. Create the Namespace object for the OpenShift sandboxed containers Operator.

   1. Create a Namespace object YAML file that contains the following manifest:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-sandboxed-containers-operator

   2. Create the Namespace object:

      $ oc create -f Namespace.yaml

2. Create the OperatorGroup object for the OpenShift sandboxed containers Operator.

   1. Create an OperatorGroup object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        targetNamespaces:
        - openshift-sandboxed-containers-operator

   2. Create the OperatorGroup object:

      $ oc create -f OperatorGroup.yaml

3. Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator.

   1. Create a Subscription object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: sandboxed-containers-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: sandboxed-containers-operator.v1.5.3

   2. Create the Subscription object:

      $ oc create -f Subscription.yaml

Procedure

1. Create a YAML file with the following manifest:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: cluster-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info

2. (Optional) If you want to install kata-remote as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: cluster-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
     kataConfigPoolSelector:
       matchLabels:
         <label_key>: '<label_value>' 1

   1

   Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported.

3. Create the KataConfig resource:

   $ oc create -f cluster-kataconfig.yaml

Procedure

1. Add runtimeClassName: kata-remote to any pod-templated object:

   • Pod objects
   • ReplicaSet objects
   • ReplicationController objects
   • StatefulSet objects
   • Deployment objects
   • DeploymentConfig objects

2. Add an annotation to the pod-templated object, defining whether to use a specific instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.

   • For a specific instance size or type, add the following annotation:

     io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>

     Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the ConfigMap for peer pods. Choose from one of those.

   • For an automatic instance size or type, add the following annotations:

     io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
     io.katacontainers.config.hypervisor.default_memory: <memory>

     Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.

   Example for Pod object

   apiVersion: v1
   kind: Pod
   metadata:
     name: hello-openshift
     labels:
       app: hello-openshift
     annotations:
       io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1
   spec:
     runtimeClassName: kata-remote
     containers:
       - name: hello-openshift
         image: quay.io/openshift/origin-hello-openshift
         ports:
           - containerPort: 8888
         securityContext:
           privileged: false
           allowPrivilegeEscalation: false
           runAsNonRoot: true
           runAsUser: 1001
           capabilities:
             drop:
               - ALL
           seccompProfile:
             type: RuntimeDefault

   1

   This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.

Procedure

1. From the Administrator perspective, navigate to Workloads → Pods.
2. Search for the pod that you want to delete using the Search by name field.
3. Click the pod name to open it.
4. On the Details page, check that kata is displayed for Runtime class.
5. Click the Actions menu and select Delete Pod.
6. Click Delete in the confirmation window.

Procedure

1. From the Administrator perspective, navigate to Operators → Installed Operators.
2. Search for the OpenShift sandboxed containers Operator using the Search by name field.
3. Click the Operator to open it, and then select the KataConfig tab.
4. Click the Options menu for the KataConfig resource, and then select Delete KataConfig.
5. Click Delete in the confirmation window.

Procedure

1. From the Administrator perspective, navigate to Operators → Installed Operators.
2. Search for the OpenShift sandboxed containers Operator using the Search by name field.
3. Click the Options menu for the Operator and select Uninstall Operator.
4. Click Uninstall in the confirmation window.

Procedure

1. From the Administrator perspective, navigate to Administration → Namespaces.
2. Search for the openshift-sandboxed-containers-operator namespace using the Search by name field.

3. Click the Options menu for the namespace and select Delete Namespace.

   Note

   If the Delete Namespace option is not available, you do not have permission to delete the namespace.

4. In the Delete Namespace pane, enter openshift-sandboxed-containers-operator and click Delete.
5. Click Delete.

Procedure

1. From the Administrator perspective, navigate to Administration → CustomResourceDefinitions.
2. Search for KataConfig using the Search by name field.
3. Click the Options menu for the KataConfig CRD, and then select Delete CustomResourceDefinition.
4. Click Delete in the confirmation window.
5. Wait for the KataConfig CRD to disappear from the list. This can take several minutes.

Procedure

1. Search for pods that use kata as the runtimeClass by running the following command:

   $ oc get pods -A -o json | jq -r '.items[] | select(.spec.runtimeClassName == "kata").metadata.name'

2. To delete each pod, run the following command:

   $ oc delete pod <pod-name>

Procedure

1. Delete the KataConfig custom resource by running the following command:

   $ oc delete kataconfig <KataConfig_CR_Name>

Procedure

1. Fetch the cluster service version (CSV) name for OpenShift sandboxed containers from the subscription by running the following command:

   CSV_NAME=$(oc get csv -n openshift-sandboxed-containers-operator -o=custom-columns=:metadata.name)

2. Delete the OpenShift sandboxed containers Operator subscription from Operator Lifecyle Manager (OLM) by running the following command:

   $ oc delete subscription sandboxed-containers-operator -n openshift-sandboxed-containers-operator

3. Delete the CSV name for OpenShift sandboxed containers by running the following command:

   $ oc delete csv ${CSV_NAME} -n openshift-sandboxed-containers-operator

4. Fetch the OpenShift sandboxed containers Operator group name by running the following command:

   $ OG_NAME=$(oc get operatorgroup -n openshift-sandboxed-containers-operator -o=jsonpath={..name})

5. Delete the OpenShift sandboxed containers Operator group name by running the following command:

   $ oc delete operatorgroup ${OG_NAME} -n openshift-sandboxed-containers-operator

6. Delete the OpenShift sandboxed containers namespace by running the following command:

   $ oc delete namespace openshift-sandboxed-containers-operator

Procedure

1. Delete the KataConfig CRD by running the following command:

   $ oc delete crd kataconfigs.kataconfiguration.openshift.io

Procedure

1. From the Administrator perspective of OpenShift Container Platform, navigate to Operators → Installed Operators.
2. Select the OpenShift sandboxed containers Operator and go to the KataConfig tab.
3. Search for the KataConfig resource using the Search by name field. The default name for the KataConfig resource is example-kataconfig.
4. Select the KataConfig resource and go to the KataConfig tab.

5. Modify the version number for kataMonitorImage:

       checkNodeEligibility: false
       kataConfigPoolSelector: null
       kataMonitorImage: 'registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel8:1.3.0'

6. Click Save.