Chapter 5. Using Container Storage Interface (CSI)

Procedure

1. Create the Pod object definition and save it to a file.

2. Embed the CSI inline ephemeral volume in the file.

   my-csi-app.yaml

   kind: Pod
   apiVersion: v1
   metadata:
     name: my-csi-app
   spec:
     containers:
       - name: my-frontend
         image: busybox
         volumeMounts:
         - mountPath: "/data"
           name: my-csi-inline-vol
         command: [ "sleep", "1000000" ]
     volumes: 1
       - name: my-csi-inline-vol
         csi:
           driver: inline.storage.kubernetes.io
           volumeAttributes:
             foo: bar

   1

   The name of the volume that is used by pods.

3. Create the object definition file that you saved in the previous step.

   $ oc create -f my-csi-app.yaml

1. Create a file with the VolumeSnapshotClass object described by the following YAML:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io 1
   deletionPolicy: Delete

   1

   The name of the CSI driver that is used to create snapshots of this VolumeSnapshotClass object. The name must be the same as the Provisioner field of the storage class that is responsible for the PVC that is being snapshotted.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshotclass.yaml

3. Create a VolumeSnapshot object:

   volumesnapshot-dynamic.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     volumeSnapshotClassName: csi-hostpath-snap 1
     source:
       persistentVolumeClaimName: myclaim 2

   1

   The request for a particular class by the volume snapshot. If the volumeSnapshotClassName setting is absent and there is a default volume snapshot class, a snapshot is created with the default volume snapshot class name. But if the field is absent and no default volume snapshot class exists, then no snapshot is created.

   2

   The name of the PersistentVolumeClaim object bound to a persistent volume. This defines what you want to create a snapshot of. Required for dynamically provisioning a snapshot.

4. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-dynamic.yaml

1. Provide a value for the volumeSnapshotContentName parameter as the source for the snapshot, in addition to defining volume snapshot class as shown above.

   volumesnapshot-manual.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: snapshot-demo
   spec:
     source:
       volumeSnapshotContentName: mycontent 1

   1

   The volumeSnapshotContentName parameter is required for pre-provisioned snapshots.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-manual.yaml

1. To display details about the volume snapshot that was created, enter the following command:

   $ oc describe volumesnapshot mysnap

   The following example displays details about the mysnap volume snapshot:

   volumesnapshot.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     source:
       persistentVolumeClaimName: myclaim
     volumeSnapshotClassName: csi-hostpath-snap
   status:
     boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 1
     creationTime: "2020-01-29T12:24:30Z" 2
     readyToUse: true 3
     restoreSize: 500Mi

   1

   The pointer to the actual storage content that was created by the controller.

   2

   The time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time.

   3

   If the value is set to true, the snapshot can be used to restore as a new PVC.
   If the value is set to false, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes.

2. To verify that the volume snapshot was created, enter the following command:

   $ oc get volumesnapshotcontent

   The pointer to the actual content is displayed. If the boundVolumeSnapshotContentName field is populated, a VolumeSnapshotContent object exists and the snapshot was created.

3. To verify that the snapshot is ready, confirm that the VolumeSnapshot object has readyToUse: true.

Procedure

1. Specify the deletion policy that you require in the VolumeSnapshotClass object, as shown in the following example:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io
   deletionPolicy: Delete 1

   1

   When deleting the volume snapshot, if the Delete value is set, the underlying snapshot is deleted along with the VolumeSnapshotContent object. If the Retain value is set, both the underlying snapshot and VolumeSnapshotContent object remain.
   If the Retain value is set and the VolumeSnapshot object is deleted without deleting the corresponding VolumeSnapshotContent object, the content remains. The snapshot itself is also retained in the storage back end.

2. Delete the volume snapshot by entering the following command:

   $ oc delete volumesnapshot <volumesnapshot_name>

   Example output

   volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted

3. If the deletion policy is set to Retain, delete the volume snapshot content by entering the following command:

   $ oc delete volumesnapshotcontent <volumesnapshotcontent_name>

4. Optional: If the VolumeSnapshot object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue:

   Important

   Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the VolumeSnapshot object. Even with the --force option, the delete operation does not delete snapshot objects until all finalizers are removed.

   $ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}'

   Example output

   volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted

   The finalizers are removed and the volume snapshot is deleted.

Procedure

1. Specify a VolumeSnapshot data source on a PVC as shown in the following:

   pvc-restore.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: myclaim-restore
   spec:
     storageClassName: csi-hostpath-sc
     dataSource:
       name: mysnap 1
       kind: VolumeSnapshot 2
       apiGroup: snapshot.storage.k8s.io 3
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   1

   Name of the VolumeSnapshot object representing the snapshot to use as source.

   2

   Must be set to the VolumeSnapshot value.

   3

   Must be set to the snapshot.storage.k8s.io value.

2. Create a PVC by entering the following command:

   $ oc create -f pvc-restore.yaml

3. Verify that the restored PVC has been created by entering the following command:

   $ oc get pvc

   A new PVC such as myclaim-restore is displayed.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-clone.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-1-clone
     namespace: mynamespace
   spec:
     storageClassName: csi-cloning 1
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
     dataSource:
       kind: PersistentVolumeClaim
       name: pvc-1

   1

   The name of the storage class that provisions the storage back end. The default storage class can be used and storageClassName can be omitted in the spec.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-clone.yaml

   A new PVC pvc-1-clone is created.

3. Verify that the volume clone was created and is ready by running the following command:

   $ oc get pvc pvc-1-clone

   The pvc-1-clone shows that it is Bound.

   You are now ready to use the newly cloned PVC to configure a pod.

4. Create and save a file with the Pod object described by the YAML. For example:

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
       - name: myfrontend
         image: dockerfile/nginx
         volumeMounts:
         - mountPath: "/var/www/html"
           name: mypd
     volumes:
       - name: mypd
         persistentVolumeClaim:
           claimName: pvc-1-clone 1

   1

   The cloned PVC created during the CSI volume cloning operation.

   The created Pod object is now ready to consume, clone, snapshot, or delete your cloned PVC independently of its original dataSource PVC.

1. Log in to the web console.

2. Install the AWS EFS CSI Operator:

   1. Click Operators OperatorHub.
   2. Locate the AWS EFS CSI Operator by typing AWS EFS CSI in the filter box.

   3. Click the AWS EFS CSI Driver Operator button.

      Important

      Be sure to select the AWS EFS CSI Driver Operator and not the AWS EFS Operator. The AWS EFS Operator is a community Operator and is not supported by Red Hat.

   4. On the AWS EFS CSI Driver Operator page, click Install.

   5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   6. Click Install.

      After the installation finishes, the AWS EFS CSI Operator is listed in the Installed Operators section of the web console.

3. Install the AWS EFS CSI Driver:

   1. Click administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, click Create ClusterCSIDriver.

   3. Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: efs.csi.aws.com
      spec:
        managementState: Managed

   4. Click Create.

   5. Wait for the following Conditions to change to a "true" status:

      • AWSEFSDriverCredentialsRequestControllerAvailable
      • AWSEFSDriverNodeServiceControllerAvailable
      • AWSEFSDriverControllerServiceControllerAvailable

1. On the AWS console, open https://console.aws.amazon.com/efs.

2. Click Create file system:

   • Enter a name for the file system.
   • For Virtual Private Cloud (VPC), select your OpenShift Container Platform’s' virtual private cloud (VPC).
   • Accept default settings for all other selections.

3. Wait for the volume and mount targets to finish being fully created:

   1. Go to https://console.aws.amazon.com/efs#/file-systems.
   2. Click your volume, and on the Network tab wait for all mount targets to become available (~1-2 minutes).
4. On the Network tab, copy the Security Group ID (you will need this in the next step).
5. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups, and find the Security Group used by the EFS volume.

6. On the Inbound rules tab, click Edit inbound rules, and then add a new rule with the following settings to allow OpenShift Container Platform nodes to access EFS volumes :

   • Type: NFS
   • Protocol: TCP
   • Port range: 2049

   • Source: Custom/IP address range of your nodes (for example: “10.0.0.0/16”)

     This step allows OpenShift Container Platform to use NFS ports from the cluster.

7. Save the rule.

1. Log in to the web console.
2. Stop all applications that use AWS EFS PVs.

3. Delete all AWS EFS PVs:

   1. Click Storage PersistentVolumeClaims.
   2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

4. Uninstall the AWS EFS CSI Driver:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.

5. Uninstall the AWS EFS CSI Operator:

   1. Click Operators Installed Operators.
   2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

Procedure

1. Enable feature gates with the TechPreviewNoUpgrade feature set (see Nodes Enabling features using feature gates).

   Important

   After turning Technology Preview features on by using feature gates, they cannot be turned off and cluster upgrades are prevented.

2. Verify the cluster operator storage:

   $ oc get co storage

   NAME    VERSION                             AVAILABLE   PROGRESSING DEGRADED    SINCE
   storage 4.9.0-0.nightly-2021-09-08-162532   True        False       False       4h26m

   • AVAILABLE should be "True".
   • PROGRESSING should be "False".
   • DEGRADED should be "False".

3. Verify the status of the pods in the openshift-cluster-csi-drivers namespace to ensure that they are running:

   $ oc get pod -n openshift-cluster-csi-drivers

   NAME                                                    READY   STATUS  RESTARTS    AGE
   azure-disk-csi-driver-controller-5949bf45fd-pm4qb       11/11   Running 0           39m
   azure-disk-csi-driver-node-2tcxr                        3/3     Running 0           53m
   azure-disk-csi-driver-node-2xjzm                        3/3     Running 0           53m
   azure-disk-csi-driver-node-6wrgk                        3/3     Running 0           53m
   azure-disk-csi-driver-node-frvx2                        3/3     Running 0           53m
   azure-disk-csi-driver-node-lf5kb                        3/3     Running 0           53m
   azure-disk-csi-driver-node-mqdhh                        3/3     Running 0           53m
   azure-disk-csi-driver-operator-7d966fc6c5-x74x5         1/1     Running 0           44m

4. Verify that the storage class is installed:

   $ oc get storageclass

   NAME                        PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE       ALLOWVOLUMEEXPANSION    AGE
   managed-premium (default)   kubernetes.io/azure-disk        Delete          WaitForFirstConsumer    true                    76m
   managed-csi                 disk.csi.azure.com              Delete          WaitForFirstConsumer    true                    51m 1

   1

   Azure storage class

1. Create a storage class with the Cloud KMS key. The following example enables dynamic provisioning of encrypted volumes:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: csi-gce-pd-cmek
   provisioner: pd.csi.storage.gke.io
   volumeBindingMode: "WaitForFirstConsumer"
   allowVolumeExpansion: true
   parameters:
     type: pd-standard
     disk-encryption-kms-key: projects/<key-project-id>/locations/<location>/keyRings/<key-ring>/cryptoKeys/<key> 1

   1

   This field must be the resource identifier for the key that will be used to encrypt new disks. Values are case-sensitive. For more information about providing key ID values, see Retrieving a resource’s ID and Getting a Cloud KMS resource ID.

   Note

   You cannot add the disk-encryption-kms-key parameter to an existing storage class. However, you can delete the storage class and recreate it with the same name and a different set of parameters. If you do this, the provisioner of the existing class must be pd.csi.storage.gke.io.

2. Deploy the storage class on your OpenShift Container Platform cluster using the oc command:

   $ oc describe storageclass csi-gce-pd-cmek

   Example output

   Name:                  csi-gce-pd-cmek
   IsDefaultClass:        No
   Annotations:           None
   Provisioner:           pd.csi.storage.gke.io
   Parameters:            disk-encryption-kms-key=projects/key-project-id/locations/location/keyRings/ring-name/cryptoKeys/key-name,type=pd-standard
   AllowVolumeExpansion:  true
   MountOptions:          none
   ReclaimPolicy:         Delete
   VolumeBindingMode:     WaitForFirstConsumer
   Events:                none

3. Create a file named pvc.yaml that matches the name of your storage class object that you created in the previous step:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: podpvc
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: csi-gce-pd-cmek
     resources:
       requests:
         storage: 6Gi

   Note

   If you marked the new storage class as default, you can omit the storageClassName field.

4. Apply the PVC on your cluster:

   $ oc apply -f pvc.yaml

5. Get the status of your PVC and verify that it is created and bound to a newly provisioned PV:

   $ oc get pvc

   Example output

   NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
   podpvc    Bound     pvc-e36abf50-84f3-11e8-8538-42010a800002   10Gi       RWO            csi-gce-pd-cmek  9s

   Note

   If your storage class has the volumeBindingMode field set to WaitForFirstConsumer, you must create a pod to use the PVC before you can verify it.

1. List the storage class:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard(default)      cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h
   standard-csi           kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h

2. Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default storage class, as shown in the following example:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

3. Make another storage class the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.

   $ oc patch storageclass standard-csi -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

4. Verify that the PVC is now referencing the CSI storage class by default:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard               kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h
   standard-csi(default)  cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h

5. Optional: You can define a new PVC without having to specify the storage class:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: cinder-claim
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   A PVC that does not specify a specific storage class is automatically provisioned by using the default storage class.

6. Optional: After the new file has been configured, create it in your cluster:

   $ oc create -f cinder-claim.yaml

1. In the OpenShift Container Platform console, click Storage Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the required options on the resulting page.

   1. Select the appropriate storage class.
   2. Enter a unique name for the storage claim.

   3. Select the access mode to specify read and write access for the PVC you are creating.

      Important

      Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

4. Define the size of the storage claim.
5. Click Create to create the persistent volume claim and generate a persistent volume.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-manila.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-manila
   spec:
     accessModes: 1
       - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     storageClassName: csi-manila-gold 2

   1

   Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

   2

   The name of the storage class that provisions the storage back end. Manila storage classes are provisioned by the Operator and have the csi-manila- prefix.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-manila.yaml

   A new PVC is created.

3. To verify that the volume was created and is ready, run the following command:

   $ oc get pvc pvc-manila

   The pvc-manila shows that it is Bound.

Procedure

1. Enable feature gates with the TechPreviewNoUpgrade feature set (see Nodes Enabling features using feature gates).

   Important

   After turning Technology Preview features on by using feature gates, they cannot be turned off and cluster upgrades are prevented.

2. Verify the cluster operator storage:

   $ oc get co storage

   NAME    VERSION                             AVAILABLE   PROGRESSING DEGRADED    SINCE
   storage 4.9.0-0.nightly-2021-09-08-162532   True        False       False       4h26m

   • AVAILABLE should be "True".
   • PROGRESSING should be "False".
   • DEGRADED should be "False".

3. Verify the status of the pods in the openshift-cluster-csi-drivers namespace to ensure that they are running:

   $ oc get pod -n openshift-cluster-csi-drivers

   NAME                                                    READY   STATUS  RESTARTS    AGE
   vmware-vsphere-csi-driver-controller-5646dbbf54-cnsx7   9/9     Running 0           4h29m
   vmware-vsphere-csi-driver-node-ch22q                    3/3     Running 0           4h37m
   vmware-vsphere-csi-driver-node-gfjrb                    3/3     Running 0           4h37m
   vmware-vsphere-csi-driver-node-ktlmp                    3/3     Running 0           4h37m
   vmware-vsphere-csi-driver-node-lgksl                    3/3     Running 0           4h37m
   vmware-vsphere-csi-driver-node-vb4gv                    3/3     Running 0           4h37m
   vmware-vsphere-csi-driver-operator-7c7fc474c-p544t      1/1     Running 0           4h29m

1. Verify that the storage class is installed:

   $ oc get storageclass

   NAME            PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE       ALLOWVOLUMEEXPANSION    AGE
   thin (default)  kubernetes.io/vsphere-volume    Delete          Immediate               false                   5h43m
   thin-csi        csi.vsphere.vmware.com          Delete          WaitForFirstConsumer    false                   4h38m 1

   1

   vSphere storage class