Chapter 7. Working with containers

Procedure

1. Create the pod for the Init Container:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: myapp-pod
        labels:
          app: myapp
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: myapp-container
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'echo The app is running! && sleep 3600']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
        initContainers:
        - name: init-myservice
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'until getent hosts myservice; do echo waiting for myservice; sleep 2; done;']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
        - name: init-mydb
          image: registry.access.redhat.com/ubi9/ubi:latest
          command: ['sh', '-c', 'until getent hosts mydb; do echo waiting for mydb; sleep 2; done;']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

   2. Create the pod:

      $ oc create -f myapp.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     0/1       Init:0/2            0          5s

      The pod status, Init:0/2, indicates it is waiting for the two services.

2. Create the myservice service.

   1. Create a YAML file similar to the following:

      kind: Service
      apiVersion: v1
      metadata:
        name: myservice
      spec:
        ports:
        - protocol: TCP
          port: 80
          targetPort: 9376

   2. Create the pod:

      $ oc create -f myservice.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     0/1       Init:1/2            0          5s

      The pod status, Init:1/2, indicates it is waiting for one service, in this case the mydb service.

3. Create the mydb service:

   1. Create a YAML file similar to the following:

      kind: Service
      apiVersion: v1
      metadata:
        name: mydb
      spec:
        ports:
        - protocol: TCP
          port: 80
          targetPort: 9377

   2. Create the pod:

      $ oc create -f mydb.yaml

   3. View the status of the pod:

      $ oc get pods

      Example output

      NAME                          READY     STATUS              RESTARTS   AGE
      myapp-pod                     1/1       Running             0          2m

      The pod status indicated that it is no longer waiting for the services and is running.

Procedure

1. To view the list of files in the volume, run the oc rsh command:

   $ oc rsh <pod>

   Example output

   sh-4.2$ ls /path/to/volume/subpath/mount
   example_file1 example_file2 example_file3

2. Specify the subPath:

   Example Pod spec with subPath parameter

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-site
   spec:
       securityContext:
         runAsNonRoot: true
         seccompProfile:
           type: RuntimeDefault
       containers:
       - name: mysql
         image: mysql
         volumeMounts:
         - mountPath: /var/lib/mysql
           name: site-data
           subPath: mysql 1
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop: [ALL]
       - name: php
         image: php
         volumeMounts:
         - mountPath: /var/www/html
           name: site-data
           subPath: html 2
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop: [ALL]
       volumes:
       - name: site-data
         persistentVolumeClaim:
           claimName: my-site-data

   1

   Databases are stored in the mysql folder.

   2

   HTML content is stored in the html folder.

1. Create the secret:

   1. Create a YAML file similar to the following, replacing the password and user information as appropriate:

      apiVersion: v1
      kind: Secret
      metadata:
        name: mysecret
      type: Opaque
      data:
        pass: MWYyZDFlMmU2N2Rm
        user: YWRtaW4=

   2. Use the following command to create the secret:

      $ oc create -f <secrets-filename>

      For example:

      $ oc create -f secret.yaml

      Example output

      secret "mysecret" created

   3. You can check that the secret was created using the following commands:

      $ oc get secret <secret-name>

      For example:

      $ oc get secret mysecret

      Example output

      NAME       TYPE      DATA      AGE
      mysecret   Opaque    2         17h

      $ oc get secret <secret-name> -o yaml

      For example:

      $ oc get secret mysecret -o yaml

      apiVersion: v1
      data:
        pass: MWYyZDFlMmU2N2Rm
        user: YWRtaW4=
      kind: Secret
      metadata:
        creationTimestamp: 2017-05-30T20:21:38Z
        name: mysecret
        namespace: default
        resourceVersion: "2107"
        selfLink: /api/v1/namespaces/default/secrets/mysecret
        uid: 959e0424-4575-11e7-9f97-fa163e4bd54c
      type: Opaque

2. Create a pod with a projected volume.

   1. Create a YAML file similar to the following, including a volumes section:

      kind: Pod
      metadata:
        name: test-projected-volume
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: test-projected-volume
          image: busybox
          args:
          - sleep
          - "86400"
          volumeMounts:
          - name: all-in-one
            mountPath: "/projected-volume"
            readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
        volumes:
        - name: all-in-one
          projected:
            sources:
            - secret:
                name: mysecret 1

      1

      The name of the secret you created.

   2. Create the pod from the configuration file:

      $ oc create -f <your_yaml_file>.yaml

      For example:

      $ oc create -f secret-pod.yaml

      Example output

      pod "test-projected-volume" created

3. Verify that the pod container is running, and then watch for changes to the pod:

   $ oc get pod <name>

   For example:

   $ oc get pod test-projected-volume

   The output should appear similar to the following:

   Example output

   NAME                    READY     STATUS    RESTARTS   AGE
   test-projected-volume   1/1       Running   0          14s

4. In another terminal, use the oc exec command to open a shell to the running container:

   $ oc exec -it <pod> <command>

   For example:

   $ oc exec -it test-projected-volume -- /bin/sh

5. In your shell, verify that the projected-volumes directory contains your projected sources:

   / # ls

   Example output

   bin               home              root              tmp
   dev               proc              run               usr
   etc               projected-volume  sys               var

Procedure

1. Create a secret to inject:

   1. Create a secret.yaml file similar to the following:

      apiVersion: v1
      kind: Secret
      metadata:
        name: mysecret
      data:
        password: <password>
        username: <username>
      type: kubernetes.io/basic-auth

   2. Create the secret object from the secret.yaml file:

      $ oc create -f secret.yaml

2. Create a pod that references the username field from the above Secret object:

   1. Create a pod.yaml file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: dapi-env-test-pod
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
          - name: env-test-container
            image: gcr.io/google_containers/busybox
            command: [ "/bin/sh", "-c", "env" ]
            env:
              - name: MY_SECRET_USERNAME
                valueFrom:
                  secretKeyRef:
                    name: mysecret
                    key: username
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
        restartPolicy: Never
      # ...

   2. Create the pod from the pod.yaml file:

      $ oc create -f pod.yaml

Procedure

1. Create a config map with the values to inject:

   1. Create a configmap.yaml file similar to the following:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: myconfigmap
      data:
        mykey: myvalue

   2. Create the config map from the configmap.yaml file:

      $ oc create -f configmap.yaml

2. Create a pod that references the above config map:

   1. Create a pod.yaml file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: dapi-env-test-pod
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
          - name: env-test-container
            image: gcr.io/google_containers/busybox
            command: [ "/bin/sh", "-c", "env" ]
            env:
              - name: MY_CONFIGMAP_VALUE
                valueFrom:
                  configMapKeyRef:
                    name: myconfigmap
                    key: mykey
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
        restartPolicy: Always
      # ...

   2. Create the pod from the pod.yaml file:

      $ oc create -f pod.yaml

Procedure

1. Create a pod that references an existing environment variable:

   1. Create a pod.yaml file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: dapi-env-test-pod
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
          - name: env-test-container
            image: gcr.io/google_containers/busybox
            command: [ "/bin/sh", "-c", "env" ]
            env:
              - name: MY_EXISTING_ENV
                value: my_value
              - name: MY_ENV_VAR_REF_ENV
                value: $(MY_EXISTING_ENV)
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
        restartPolicy: Never
      # ...

   2. Create the pod from the pod.yaml file:

      $ oc create -f pod.yaml

Procedure

1. Create a pod that references an existing environment variable:

   1. Create a pod.yaml file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: dapi-env-test-pod
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
          - name: env-test-container
            image: gcr.io/google_containers/busybox
            command: [ "/bin/sh", "-c", "env" ]
            env:
              - name: MY_NEW_ENV
                value: $$(SOME_OTHER_ENV)
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
        restartPolicy: Never
      # ...

   2. Create the pod from the pod.yaml file:

      $ oc create -f pod.yaml

Procedure

1. View the existing predefined list by running the following command:

   $ oc get cm -n openshift-multus cni-sysctl-allowlist -oyaml

   Expected output

   apiVersion: v1
   data:
     allowlist.conf: |-
       ^net.ipv4.conf.IFNAME.accept_redirects$
       ^net.ipv4.conf.IFNAME.accept_source_route$
       ^net.ipv4.conf.IFNAME.arp_accept$
       ^net.ipv4.conf.IFNAME.arp_notify$
       ^net.ipv4.conf.IFNAME.disable_policy$
       ^net.ipv4.conf.IFNAME.secure_redirects$
       ^net.ipv4.conf.IFNAME.send_redirects$
       ^net.ipv6.conf.IFNAME.accept_ra$
       ^net.ipv6.conf.IFNAME.accept_redirects$
       ^net.ipv6.conf.IFNAME.accept_source_route$
       ^net.ipv6.conf.IFNAME.arp_accept$
       ^net.ipv6.conf.IFNAME.arp_notify$
       ^net.ipv6.neigh.IFNAME.base_reachable_time_ms$
       ^net.ipv6.neigh.IFNAME.retrans_time_ms$
   kind: ConfigMap
   metadata:
     annotations:
       kubernetes.io/description: |
         Sysctl allowlist for nodes.
       release.openshift.io/version: 4.17.0-0.nightly-2022-11-16-003434
     creationTimestamp: "2022-11-17T14:09:27Z"
     name: cni-sysctl-allowlist
     namespace: openshift-multus
     resourceVersion: "2422"
     uid: 96d138a3-160e-4943-90ff-6108fa7c50c3

2. Edit the list by using the following command:

   $ oc edit cm -n openshift-multus cni-sysctl-allowlist -oyaml

   For example, to allow you to be able to implement stricter reverse path forwarding you need to add ^net.ipv4.conf.IFNAME.rp_filter$ and ^net.ipv6.conf.IFNAME.rp_filter$ to the list as shown here:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: v1
   data:
     allowlist.conf: |-
       ^net.ipv4.conf.IFNAME.accept_redirects$
       ^net.ipv4.conf.IFNAME.accept_source_route$
       ^net.ipv4.conf.IFNAME.arp_accept$
       ^net.ipv4.conf.IFNAME.arp_notify$
       ^net.ipv4.conf.IFNAME.disable_policy$
       ^net.ipv4.conf.IFNAME.secure_redirects$
       ^net.ipv4.conf.IFNAME.send_redirects$
       ^net.ipv4.conf.IFNAME.rp_filter$
       ^net.ipv6.conf.IFNAME.accept_ra$
       ^net.ipv6.conf.IFNAME.accept_redirects$
       ^net.ipv6.conf.IFNAME.accept_source_route$
       ^net.ipv6.conf.IFNAME.arp_accept$
       ^net.ipv6.conf.IFNAME.arp_notify$
       ^net.ipv6.neigh.IFNAME.base_reachable_time_ms$
       ^net.ipv6.neigh.IFNAME.retrans_time_ms$
       ^net.ipv6.conf.IFNAME.rp_filter$

3. Save the changes to the file and exit.

   Note

   The removal of sysctls is also supported. Edit the file, remove the sysctl or sysctls then save the changes and exit.

1. Create a network attachment definition, such as reverse-path-fwd-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: tuningnad
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "tuningnad",
       "plugins": [{
         "type": "bridge"
         },
         {
         "type": "tuning",
         "sysctl": {
            "net.ipv4.conf.IFNAME.rp_filter": "1"
           }
       }
     ]
   }'

2. Apply the yaml by running the following command:

   $ oc apply -f reverse-path-fwd-example.yaml

   Example output

   networkattachmentdefinition.k8.cni.cncf.io/tuningnad created

3. Create a pod such as examplepod.yaml using the following YAML:

   apiVersion: v1
   kind: Pod
   metadata:
     name: example
     labels:
       app: httpd
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: tuningnad  1
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
       - name: httpd
         image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest'
         ports:
           - containerPort: 8080
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop:
               - ALL

   1

   Specify the name of the configured NetworkAttachmentDefinition.

4. Apply the yaml by running the following command:

   $ oc apply -f examplepod.yaml

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   example   1/1     Running   0          47s

6. Log in to the pod by running the following command:

   $ oc rsh example

7. Verify the value of the configured sysctl flag. For example, find the value net.ipv4.conf.net1.rp_filter by running the following command:

   sh-4.4# sysctl net.ipv4.conf.net1.rp_filter

   Expected output

   net.ipv4.conf.net1.rp_filter = 1

Procedure

1. Create a YAML file sysctl_pod.yaml that defines an example pod and add the securityContext spec, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example
     namespace: default
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 1
         runAsGroup: 3000 2
         allowPrivilegeEscalation: false 3
         capabilities: 4
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 5
       seccompProfile: 6
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "1"
       - name: net.ipv4.ip_local_port_range
         value: "32770       60666"
       - name: net.ipv4.tcp_syncookies
         value: "0"
       - name: net.ipv4.ping_group_range
         value: "0           200000000"

   1

   runAsUser controls which user ID the container is run with.

   2

   runAsGroup controls which primary group ID the containers is run with.

   3

   allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   4

   capabilities permit privileged actions without giving full root access. This policy ensures all capabilities are dropped from the pod.

   5

   runAsNonRoot: true requires that the container will run with a user with any UID other than 0.

   6

   RuntimeDefault enables the default seccomp profile for a pod or container workload.

2. Create the pod by running the following command:

   $ oc apply -f sysctl_pod.yaml

3. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME              READY   STATUS            RESTARTS   AGE
   sysctl-example    1/1     Running           0          14s

4. Log in to the pod by running the following command:

   $ oc rsh sysctl-example

5. Verify the values of the configured sysctl flags. For example, find the value kernel.shm_rmid_forced by running the following command:

   sh-4.4# sysctl kernel.shm_rmid_forced

   Expected output

   kernel.shm_rmid_forced = 1

Procedure

1. Create a YAML file sysctl-example-unsafe.yaml that defines an example pod and add the securityContext specification, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example-unsafe
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000
         runAsGroup: 3000
         allowPrivilegeEscalation: false
         capabilities:
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "0"
       - name: net.core.somaxconn
         value: "1024"
       - name: kernel.msgmax
         value: "65536"

2. Create the pod using the following command:

   $ oc apply -f sysctl-example-unsafe.yaml

3. Verify that the pod is scheduled but does not deploy because unsafe sysctls are not allowed for the node using the following command:

   $ oc get pod

   Example output

   NAME                       READY             STATUS            RESTARTS   AGE
   sysctl-example-unsafe      0/1               SysctlForbidden   0          14s

Procedure

1. List existing MachineConfig objects for your OpenShift Container Platform cluster to decide how to label your machine config by running the following command:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-bfb92f0cd1684e54d8e234ab7423cc96   True      False      False      3              3                   3                     0                      42m
   worker   rendered-worker-21b6cb9a0f8919c88caf39db80ac1fce   True      False      False      3              3                   3                     0                      42m

2. Add a label to the machine config pool where the containers with the unsafe sysctls will run by running the following command:

   $ oc label machineconfigpool worker custom-kubelet=sysctl

3. Create a YAML file set-sysctl-worker.yaml that defines a KubeletConfig custom resource (CR):

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-kubelet
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: sysctl 1
     kubeletConfig:
       allowedUnsafeSysctls: 2
         - "kernel.msg*"
         - "net.core.somaxconn"

   1

   Specify the label from the machine config pool.

   2

   List the unsafe sysctls you want to allow.

4. Create the object by running the following command:

   $ oc apply -f set-sysctl-worker.yaml

5. Wait for the Machine Config Operator to generate the new rendered configuration and apply it to the machines by running the following command:

   $ oc get machineconfigpool worker -w

   After some minutes the UPDATING status changes from True to False:

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker   rendered-worker-f1704a00fc6f30d3a7de9a15fd68a800   False     True       False      3              2                   2                     0                      71m
   worker   rendered-worker-f1704a00fc6f30d3a7de9a15fd68a800   False     True       False      3              2                   3                     0                      72m
   worker   rendered-worker-0188658afe1f3a183ec8c4f14186f4d5   True      False      False      3              3                   3                     0                      72m

6. Create a YAML file sysctl-example-safe-unsafe.yaml that defines an example pod and add the securityContext spec, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example-safe-unsafe
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000
         runAsGroup: 3000
         allowPrivilegeEscalation: false
         capabilities:
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "0"
       - name: net.core.somaxconn
         value: "1024"
       - name: kernel.msgmax
         value: "65536"

7. Create the pod by running the following command:

   $ oc apply -f sysctl-example-safe-unsafe.yaml

   Expected output

   Warning: would violate PodSecurity "restricted:latest": forbidden sysctls (net.core.somaxconn, kernel.msgmax)
   pod/sysctl-example-safe-unsafe created

8. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME                         READY   STATUS    RESTARTS   AGE
   sysctl-example-safe-unsafe   1/1     Running   0          19s

9. Log in to the pod by running the following command:

   $ oc rsh sysctl-example-safe-unsafe

10. Verify the values of the configured sysctl flags. For example, find the value net.core.somaxconn by running the following command:

    sh-4.4# sysctl net.core.somaxconn

    Expected output

    net.core.somaxconn = 1024

Procedure

1. Create the pod.

   $ oc exec -ti no-priv -- /bin/bash

   $ cat >> Dockerfile <<EOF
   FROM registry.access.redhat.com/ubi9
   EOF

   $ podman build .

2. Implement /dev/fuse by adding the /dev/fuse device to the io.kubernetes.cri-o.Devices annotation.

   io.kubernetes.cri-o.Devices: "/dev/fuse"

   For example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: podman-pod
     annotations:
       io.kubernetes.cri-o.Devices: "/dev/fuse"

3. Configure the /dev/fuse device in your pod specifications.

   spec:
     containers:
     - name: podman-container
       image: quay.io/podman/stable
       args:
       - sleep
       - "1000000"
       securityContext:
         runAsUser: 1000