Chapter 5. Using Container Storage Interface (CSI)

Procedure

1. Create the Pod object definition and save it to a file.

2. Embed the CSI inline ephemeral volume in the file.

   my-csi-app.yaml

   kind: Pod
   apiVersion: v1
   metadata:
     name: my-csi-app
   spec:
     containers:
       - name: my-frontend
         image: busybox
         volumeMounts:
         - mountPath: "/data"
           name: my-csi-inline-vol
         command: [ "sleep", "1000000" ]
     volumes: 1
       - name: my-csi-inline-vol
         csi:
           driver: inline.storage.kubernetes.io
           volumeAttributes:
             foo: bar

   1

   The name of the volume that is used by pods.

3. Create the object definition file that you saved in the previous step.

   $ oc create -f my-csi-app.yaml

Procedure

1. Grant a given service account RBAC permissions to use the SharedSecret CR instance in its pod by using oc apply with YAML content:

   Note

   Currently, kubectl and oc have hard-coded special case logic restricting the use verb to roles centered around pod security. Therefore, you cannot use oc create role …​ to create the role needed for consuming SharedSecret CR instances.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedsecrets
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the RoleBinding associated with the role by using the oc command:

   $ oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the SharedSecret CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedSecret: my-share
   
   EOF

Procedure

1. Create a SharedConfigMap CR instance for the config map that you want to share across namespaces in the cluster:

   $ oc apply -f - <<EOF
   apiVersion: sharedresource.openshift.io/v1alpha1
   kind: SharedConfigMap
   metadata:
     name: my-share
   spec:
     configMapRef:
       name: <name of configmap>
       namespace: <namespace of configmap>
   EOF

Procedure

1. Grant a given service account RBAC permissions to use the SharedConfigMap CR instance in its pod by using oc apply with YAML content.

   Note

   Currently, kubectl and oc have hard-coded special case logic restricting the use verb to roles centered around pod security. Therefore, you cannot use oc create role …​ to create the role needed for consuming a SharedConfigMap CR instance.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedconfigmaps
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the RoleBinding associated with the role by using the oc command:

   oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the SharedConfigMap CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedConfigMap: my-share
   
   EOF

1. Create a file with the VolumeSnapshotClass object described by the following YAML:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io 1
   deletionPolicy: Delete

   1

   The name of the CSI driver that is used to create snapshots of this VolumeSnapshotClass object. The name must be the same as the Provisioner field of the storage class that is responsible for the PVC that is being snapshotted.

   Note

   Depending on the driver that you used to configure persistent storage, additional parameters might be required. You can also use an existing VolumeSnapshotClass object.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshotclass.yaml

3. Create a VolumeSnapshot object:

   volumesnapshot-dynamic.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     volumeSnapshotClassName: csi-hostpath-snap 1
     source:
       persistentVolumeClaimName: myclaim 2

   1

   The request for a particular class by the volume snapshot. If the volumeSnapshotClassName setting is absent and there is a default volume snapshot class, a snapshot is created with the default volume snapshot class name. But if the field is absent and no default volume snapshot class exists, then no snapshot is created.

   2

   The name of the PersistentVolumeClaim object bound to a persistent volume. This defines what you want to create a snapshot of. Required for dynamically provisioning a snapshot.

4. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-dynamic.yaml

1. Provide a value for the volumeSnapshotContentName parameter as the source for the snapshot, in addition to defining volume snapshot class as shown above.

   volumesnapshot-manual.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: snapshot-demo
   spec:
     source:
       volumeSnapshotContentName: mycontent 1

   1

   The volumeSnapshotContentName parameter is required for pre-provisioned snapshots.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-manual.yaml

1. To display details about the volume snapshot that was created, enter the following command:

   $ oc describe volumesnapshot mysnap

   The following example displays details about the mysnap volume snapshot:

   volumesnapshot.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     source:
       persistentVolumeClaimName: myclaim
     volumeSnapshotClassName: csi-hostpath-snap
   status:
     boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 1
     creationTime: "2020-01-29T12:24:30Z" 2
     readyToUse: true 3
     restoreSize: 500Mi

   1

   The pointer to the actual storage content that was created by the controller.

   2

   The time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time.

   3

   If the value is set to true, the snapshot can be used to restore as a new PVC.
   If the value is set to false, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes.

2. To verify that the volume snapshot was created, enter the following command:

   $ oc get volumesnapshotcontent

   The pointer to the actual content is displayed. If the boundVolumeSnapshotContentName field is populated, a VolumeSnapshotContent object exists and the snapshot was created.

3. To verify that the snapshot is ready, confirm that the VolumeSnapshot object has readyToUse: true.

Procedure

1. Specify the deletion policy that you require in the VolumeSnapshotClass object, as shown in the following example:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io
   deletionPolicy: Delete 1

   1

   When deleting the volume snapshot, if the Delete value is set, the underlying snapshot is deleted along with the VolumeSnapshotContent object. If the Retain value is set, both the underlying snapshot and VolumeSnapshotContent object remain.
   If the Retain value is set and the VolumeSnapshot object is deleted without deleting the corresponding VolumeSnapshotContent object, the content remains. The snapshot itself is also retained in the storage back end.

2. Delete the volume snapshot by entering the following command:

   $ oc delete volumesnapshot <volumesnapshot_name>

   Example output

   volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted

3. If the deletion policy is set to Retain, delete the volume snapshot content by entering the following command:

   $ oc delete volumesnapshotcontent <volumesnapshotcontent_name>

4. Optional: If the VolumeSnapshot object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue:

   Important

   Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the VolumeSnapshot object. Even with the --force option, the delete operation does not delete snapshot objects until all finalizers are removed.

   $ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}'

   Example output

   volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted

   The finalizers are removed and the volume snapshot is deleted.

Procedure

1. Specify a VolumeSnapshot data source on a PVC as shown in the following:

   pvc-restore.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: myclaim-restore
   spec:
     storageClassName: csi-hostpath-sc
     dataSource:
       name: mysnap 1
       kind: VolumeSnapshot 2
       apiGroup: snapshot.storage.k8s.io 3
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   1

   Name of the VolumeSnapshot object representing the snapshot to use as source.

   2

   Must be set to the VolumeSnapshot value.

   3

   Must be set to the snapshot.storage.k8s.io value.

2. Create a PVC by entering the following command:

   $ oc create -f pvc-restore.yaml

3. Verify that the restored PVC has been created by entering the following command:

   $ oc get pvc

   A new PVC such as myclaim-restore is displayed.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-clone.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-1-clone
     namespace: mynamespace
   spec:
     storageClassName: csi-cloning 1
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
     dataSource:
       kind: PersistentVolumeClaim
       name: pvc-1

   1

   The name of the storage class that provisions the storage back end. The default storage class can be used and storageClassName can be omitted in the spec.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-clone.yaml

   A new PVC pvc-1-clone is created.

3. Verify that the volume clone was created and is ready by running the following command:

   $ oc get pvc pvc-1-clone

   The pvc-1-clone shows that it is Bound.

   You are now ready to use the newly cloned PVC to configure a pod.

4. Create and save a file with the Pod object described by the YAML. For example:

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
       - name: myfrontend
         image: dockerfile/nginx
         volumeMounts:
         - mountPath: "/var/www/html"
           name: mypd
     volumes:
       - name: mypd
         persistentVolumeClaim:
           claimName: pvc-1-clone 1

   1

   The cloned PVC created during the CSI volume cloning operation.

   The created Pod object is now ready to consume, clone, snapshot, or delete your cloned PVC independently of its original dataSource PVC.

1. Log in to the web console.
2. Click Administration > CustomResourceDefinitions.
3. On the CustomResourceDefinitions page, type clustercsidriver to find the ClusterCSIDriver object.
4. Click ClusterCSIDriver, and then click the Instances tab.
5. Click the name of the desired instance, and then click the YAML tab.

6. Add the spec.storageClassState field with a value of Managed, Unmanaged, or Removed.

   Example

   ...
   spec:
     driverConfig:
       driverType: ''
     logLevel: Normal
     managementState: Managed
     observedConfig: null
     operatorLogLevel: Normal
     storageClassState: Unmanaged 1
   ...

   1

   spec.storageClassState field set to "Unmanaged"

7. Click Save.

1. List the storage classes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3 (default)        kubernetes.io/aws-ebs 1
   standard             kubernetes.io/aws-ebs

   1

   (default) indicates the default storage class.

2. Make the desired storage class the default.

   For the desired storage class, set the storageclass.kubernetes.io/is-default-class annotation to true by running the following command:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

   Note

   You can have multiple default storage classes for a short time. However, you should ensure that only one default storage class exists eventually.

   With multiple default storage classes present, any persistent volume claim (PVC) requesting the default storage class (pvc.spec.storageClassName=nil) gets the most recently created default storage class, regardless of the default status of that storage class, and the administrator receives an alert in the alerts dashboard that there are multiple default storage classes, MultipleDefaultStorageClasses.

3. Remove the default storage class setting from the old default storage class.

   For the old default storage class, change the value of the storageclass.kubernetes.io/is-default-class annotation to false by running the following command:

   $ oc patch storageclass gp3 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

4. Verify the changes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3                  kubernetes.io/aws-ebs
   standard (default)   kubernetes.io/aws-ebs

1. After a node is detected as unhealthy, shut down the worker node.

2. Ensure that the node is shutdown by running the following command and checking the status:

   oc get node <node name> 1

   1

   <node name> = name of the non-gracefully shutdown node

   Important

   If the node is not completely shut down, do not proceed with tainting the node. If the node is still up and the taint is applied, filesystem corruption can occur.

3. Taint the corresponding node object by running the following command:

   oc adm taint node <node name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute 1

   1

   <node name> = name of the non-gracefully shutdown node

   After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node.

   Example

   The resulting YAML file resembles the following:

   spec:
     taints:
     - effect: NoExecute
       key: node.kubernetes.io/out-of-service
       value: nodeshutdown

4. Restart the node.
5. Remove the taint.

1. If you are using AWS EFS with AWS Secure Token Service (STS), obtain a role Amazon Resource Name (ARN) for STS. This is required for installing the AWS EFS CSI Driver Operator.
2. Install the AWS EFS CSI Driver Operator.
3. Install the AWS EFS CSI Driver.

1. Install OpenShift Container Platform cluster with AWS account A and install the EFS CSI Driver Operator.

2. Create an EFS volume in AWS account B:

   1. Create a virtual private cloud (VPC) called, for example, "my-efs-vpc” with CIDR, for example, “172.20.0.0/16” and subnet for the AWS EFS volume.
   2. On the AWS console, go to https://console.aws.amazon.com/efs.

   3. Click Create new filesystem:

      1. Create a filesystem named, for example, "my-filesystem”.
      2. Select the VPC created earlier (“my-efs-vpc”).
      3. Accept the default for the remaining settings.

   4. Ensure that the volume and Mount Targets have been created:

      1. Check https://console.aws.amazon.com/efs#/file-systems.
      2. Click your volume, and on the Network tab wait for all Mount Targets to be available (approximately 1-2 minutes).
   5. On the Network tab, copy the Security Group ID. You will need it for the next step.

3. Configure networking access to the AWS EFS volume on AWS account B:

   1. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups.
   2. Find the Security Group used by the AWS EFS volume by filtering for the group ID copied earlier.

   3. On the Inbound rules tab, click Edit inbound rules, and then add a new rule to allow OpenShift Container Platform nodes to access the AWS EFS volumes (that is, use NFS ports from the cluster):

      • Type: NFS
      • Protocol: TCP
      • Port range: 2049
      • Source: Custom/IP address range of your OpenShift Container Platform cluster nodes (for example, “10.0.0.0/16”)

   4. Save the rule.

      Note

      If you encounter mounting issues, re-check the port number, IP address range, and verify that the AWS EFS volume uses the expected security group.

4. Create VPC peering between the OpenShift Container Platform cluster VPC in AWS account A and the AWS EFS VPC in AWS account B:

   Ensure the two VPCs are using different network CIDRs, and after creating the VPC peering, add routes in each VPC to connect the two VPC networks.

   1. Create a peering connection called, for example, “my-efs-crossaccount-peering-connection” in account B. For the local VPC ID, use the EFS-located VPC. To peer with the VPC for account A, for the VPC ID use the OpenShift Container Platform cluster VPC ID.
   2. Accept the peer connection in AWS account A.

   3. Modify the route table of each subnet (EFS-volume used subnets) in AWS account B:

      1. On the left pane, under Virtual private cloud, click the down arrow to expand the available options.
      2. Under Virtual private cloud, click Route tables".
      3. Click the Routes tab.
      4. Under Destination, enter 10.0.0.0/16.
      5. Under Target, use the peer connection type point from the created peer connection.

   4. Modify the route table of each subnet (OpenShift Container Platform cluster nodes used subnets) in AWS account A:

      1. On the left pane, under Virtual private cloud, click the down arrow to expand the available options.
      2. Under Virtual private cloud, click Route tables".
      3. Click the Routes tab.
      4. Under Destination, enter the CIDR for the VPC in account B, which for this example is 172.20.0.0/16.
      5. Under Target, use the peer connection type point from the created peer connection.

5. Create an IAM role, for example, “my-efs-acrossaccount-role” in AWS account B, which has a trust relationship with AWS account A, and add an inline AWS EFS policy with permissions to call “my-efs-acrossaccount-driver-policy”.

   This role is used by the CSI driver’s controller service running on the OpenShift Container Platform cluster in AWS account A to determine the mount targets for your file system in AWS account B.

   # Trust relationships trusted entity trusted account A configuration on my-efs-acrossaccount-role in account B
   
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::301721915996:root"
               },
               "Action": "sts:AssumeRole",
               "Condition": {}
           }
       ]
   }
   
   # my-cross-account-assume-policy policy attached to my-efs-acrossaccount-role in account B
   
   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Action": "sts:AssumeRole",
           "Resource": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
       }
   }
   
   # my-efs-acrossaccount-driver-policy attached to my-efs-acrossaccount-role in account B
   
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeNetworkInterfaces",
                   "ec2:DescribeSubnets"
               ],
               "Resource": "*"
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": [
                   "elasticfilesystem:DescribeMountTargets",
                   "elasticfilesystem:DeleteAccessPoint",
                   "elasticfilesystem:ClientMount",
                   "elasticfilesystem:DescribeAccessPoints",
                   "elasticfilesystem:ClientWrite",
                   "elasticfilesystem:ClientRootAccess",
                   "elasticfilesystem:DescribeFileSystems",
                   "elasticfilesystem:CreateAccessPoint"
               ],
               "Resource": [
                   "arn:aws:elasticfilesystem:*:589722580343:access-point/*",
                   "arn:aws:elasticfilesystem:*:589722580343:file-system/*"
               ]
           }
       ]
   }

6. In AWS account A, attach an inline policy to the IAM role of the AWS EFS CSI driver’s controller service account with the necessary permissions to perform Security Token Service (STS) assume role on the IAM role created earlier.

   # my-cross-account-assume-policy policy attached to Openshift cluster efs csi driver user in account A
   
   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Action": "sts:AssumeRole",
           "Resource": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
       }
   }

7. In AWS account A, attach the AWS-managed policy “AmazonElasticFileSystemClientFullAccess” to OpenShift Container Platform cluster master role. The role name is in the form <clusterID>-master-role (for example, my-0120ef-czjrl-master-role).

8. Create a Kubernetes secret with awsRoleArn as the key and the role created earlier as the value:

   $ oc -n openshift-cluster-csi-drivers create secret generic my-efs-cross-account --from-literal=awsRoleArn='arn:aws:iam::589722580343:role/my-efs-acrossaccount-role'

   Since the driver controller needs to get the cross account role information from the secret, you need to add the secret role binding to the AWS EFS CSI driver controller ServiceAccount (SA):

   $ oc -n openshift-cluster-csi-drivers create role access-secrets --verb=get,list,watch --resource=secrets
   
   $ oc -n openshift-cluster-csi-drivers create rolebinding --role=access-secrets default-to-secrets --serviceaccount=openshift-cluster-csi-drivers:aws-efs-csi-driver-controller-sa

9. Create a filesystem policy for the file system (AWS EFS volume) in account B, which allows AWS account A to perform a mount on it.

   This step is not mandatory, but can be safer for AWS EFS volume usage.

   # EFS volume filesystem policy in account B
   {
       "Version": "2012-10-17",
       "Id": "efs-policy-wizard-8089bf4a-9787-40f0-958e-bc2363012ace",
       "Statement": [
           {
               "Sid": "efs-statement-bd285549-cfa2-4f8b-861e-c372399fd238",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "*"
               },
               "Action": [
                   "elasticfilesystem:ClientRootAccess",
                   "elasticfilesystem:ClientWrite",
                   "elasticfilesystem:ClientMount"
               ],
               "Resource": "arn:aws:elasticfilesystem:us-east-2:589722580343:file-system/fs-091066a9bf9becbd5",
               "Condition": {
                   "Bool": {
                       "elasticfilesystem:AccessedViaMountTarget": "true"
                   }
               }
           },
           {
               "Sid": "efs-statement-03646e39-d80f-4daf-b396-281be1e43bab",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
               },
               "Action": [
                   "elasticfilesystem:ClientRootAccess",
                   "elasticfilesystem:ClientWrite",
                   "elasticfilesystem:ClientMount"
               ],
               "Resource": "arn:aws:elasticfilesystem:us-east-2:589722580343:file-system/fs-091066a9bf9becbd5"
           }
       ]
   }

10. Create an AWS EFS volume storage class using a similar configuration to the following:

    # The cross account efs volume storageClass
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: efs-cross-account-mount-sc
    provisioner: efs.csi.aws.com
    mountOptions:
      - tls
    parameters:
      provisioningMode: efs-ap
      fileSystemId: fs-00f6c3ae6f06388bb
      directoryPerms: "700"
      gidRangeStart: "1000"
      gidRangeEnd: "2000"
      basePath: "/account-a-data"
      csi.storage.k8s.io/provisioner-secret-name: my-efs-cross-account
      csi.storage.k8s.io/provisioner-secret-namespace: openshift-cluster-csi-drivers
    volumeBindingMode: Immediate

1. On the AWS console, open https://console.aws.amazon.com/efs.

2. Click Create file system:

   • Enter a name for the file system.
   • For Virtual Private Cloud (VPC), select your OpenShift Container Platform’s' virtual private cloud (VPC).
   • Accept default settings for all other selections.

3. Wait for the volume and mount targets to finish being fully created:

   1. Go to https://console.aws.amazon.com/efs#/file-systems.
   2. Click your volume, and on the Network tab wait for all mount targets to become available (~1-2 minutes).
4. On the Network tab, copy the Security Group ID (you will need this in the next step).
5. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups, and find the Security Group used by the EFS volume.

6. On the Inbound rules tab, click Edit inbound rules, and then add a new rule with the following settings to allow OpenShift Container Platform nodes to access EFS volumes :

   • Type: NFS
   • Protocol: TCP
   • Port range: 2049

   • Source: Custom/IP address range of your nodes (for example: “10.0.0.0/16”)

     This step allows OpenShift Container Platform to use NFS ports from the cluster.

7. Save the rule.

1. Log in to the web console.
2. Stop all applications that use AWS EFS PVs.

3. Delete all AWS EFS PVs:

   1. Click Storage PersistentVolumeClaims.
   2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

4. Uninstall the AWS EFS CSI driver:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click Administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.

5. Uninstall the AWS EFS CSI Operator:

   1. Click Operators Installed Operators.
   2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

1. Create a storage class designating the storage account type using a YAML file similar to the following:

   $ oc create -f - << EOF
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <storage-class> 1
   provisioner: disk.csi.azure.com
   parameters:
     skuName: <storage-class-account-type> 2
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   EOF

   1

   Storage class name.

   2

   Storage account type. This corresponds to your Azure storage account SKU tier:`Standard_LRS`, Premium_LRS, StandardSSD_LRS, UltraSSD_LRS, Premium_ZRS, StandardSSD_ZRS, PremiumV2_LRS.

   Note

   For PremiumV2_LRS, specify cachingMode: None in storageclass.parameters.

2. Ensure that the storage class was created by listing the storage classes:

   $ oc get storageclass

   Example output

   $ oc get storageclass
   NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   68m
   managed-csi (default)   disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   68m
   sc-prem-zrs             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   4m25s 1

   1

   New storage class with storage account type.

1. Create a storage class with the Cloud KMS key. The following example enables dynamic provisioning of encrypted volumes:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: csi-gce-pd-cmek
   provisioner: pd.csi.storage.gke.io
   volumeBindingMode: "WaitForFirstConsumer"
   allowVolumeExpansion: true
   parameters:
     type: pd-standard
     disk-encryption-kms-key: projects/<key-project-id>/locations/<location>/keyRings/<key-ring>/cryptoKeys/<key> 1

   1

   This field must be the resource identifier for the key that will be used to encrypt new disks. Values are case-sensitive. For more information about providing key ID values, see Retrieving a resource’s ID and Getting a Cloud KMS resource ID.

   Note

   You cannot add the disk-encryption-kms-key parameter to an existing storage class. However, you can delete the storage class and recreate it with the same name and a different set of parameters. If you do this, the provisioner of the existing class must be pd.csi.storage.gke.io.

2. Deploy the storage class on your OpenShift Container Platform cluster using the oc command:

   $ oc describe storageclass csi-gce-pd-cmek

   Example output

   Name:                  csi-gce-pd-cmek
   IsDefaultClass:        No
   Annotations:           None
   Provisioner:           pd.csi.storage.gke.io
   Parameters:            disk-encryption-kms-key=projects/key-project-id/locations/location/keyRings/ring-name/cryptoKeys/key-name,type=pd-standard
   AllowVolumeExpansion:  true
   MountOptions:          none
   ReclaimPolicy:         Delete
   VolumeBindingMode:     WaitForFirstConsumer
   Events:                none

3. Create a file named pvc.yaml that matches the name of your storage class object that you created in the previous step:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: podpvc
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: csi-gce-pd-cmek
     resources:
       requests:
         storage: 6Gi

   Note

   If you marked the new storage class as default, you can omit the storageClassName field.

4. Apply the PVC on your cluster:

   $ oc apply -f pvc.yaml

5. Get the status of your PVC and verify that it is created and bound to a newly provisioned PV:

   $ oc get pvc

   Example output

   NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
   podpvc    Bound     pvc-e36abf50-84f3-11e8-8538-42010a800002   10Gi       RWO            csi-gce-pd-cmek  9s

   Note

   If your storage class has the volumeBindingMode field set to WaitForFirstConsumer, you must create a pod to use the PVC before you can verify it.

1. Log in to the web console.

2. Enable the Filestore API in the GCE project by running the following command:

   $ gcloud services enable file.googleapis.com  --project <my_gce_project> 1

   1

   Replace <my_gce_project> with your Google Cloud project.

   You can also do this using Google Cloud web console.

3. Install the GCP Filestore CSI Operator:

   1. Click Operators OperatorHub.
   2. Locate the GCP Filestore CSI Operator by typing GCP Filestore in the filter box.
   3. Click the GCP Filestore CSI Driver Operator button.
   4. On the GCP Filestore CSI Driver Operator page, click Install.

   5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   6. Click Install.

      After the installation finishes, the GCP Filestore CSI Operator is listed in the Installed Operators section of the web console.

4. Install the GCP Filestore CSI Driver:

   1. Click administration CustomResourceDefinitions ClusterCSIDriver.

   2. On the Instances tab, click Create ClusterCSIDriver.

      Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: filestore.csi.storage.gke.io
      spec:
        managementState: Managed

   3. Click Create.

   4. Wait for the following Conditions to change to a "true" status:

      • GCPFilestoreDriverCredentialsRequestControllerAvailable
      • GCPFilestoreDriverNodeServiceControllerAvailable
      • GCPFilestoreDriverControllerServiceControllerAvailable

1. Create a storage class using the following example YAML file:

   Example YAML file

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: filestore-csi
   provisioner: filestore.csi.storage.gke.io
   parameters:
     connect-mode: DIRECT_PEERING 1
     network: network-name 2
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer

   1

   For a shared VPC, use the connect-mode parameter set to PRIVATE_SERVICE_ACCESS. For a non-shared VPC, the value is DIRECT_PEERING, which is the default setting.

   2

   Specify the name of the GCP virtual private cloud (VPC) network where Filestore instances should be created in.

2. Specify the name of the VPC network where Filestore instances should be created in.

   It is recommended to specify the VPC network that the Filestore instances should be created in. If no VPC network is specified, the Container Storage Interface (CSI) driver tries to create the instances in the default VPC network of the project.

   On IPI installations, the VPC network name is typically the cluster name with the suffix "-network". However, on UPI installations, the VPC network name can be any value chosen by the user.

   For a shared VPC (connect-mode = PRIVATE_SERVICE_ACCESS), the network needs to be the full VPC name. For example: projects/shared-vpc-name/global/networks/gcp-filestore-network.

   You can find out the VPC network name by inspecting the MachineSets objects with the following command:

   $ oc -n openshift-machine-api get machinesets -o yaml | grep "network:"
               - network: gcp-filestore-network
   (...)

   In this example, the VPC network name in this cluster is "gcp-filestore-network".

1. List all PVCs that were created using the storage class filestore-csi:

   $ oc get pvc -o json -A | jq -r '.items[] | select(.spec.storageClassName == "filestore-csi")

2. Delete all of the PVCs listed by the previous command:

   $ oc delete <pvc-name> 1

   1

   Replace <pvc-name> with the name of any PVC that you need to delete.

1. List the storage class:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard(default)      cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h
   standard-csi           kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h

2. Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default storage class, as shown in the following example:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

3. Make another storage class the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.

   $ oc patch storageclass standard-csi -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

4. Verify that the PVC is now referencing the CSI storage class by default:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard               kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h
   standard-csi(default)  cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h

5. Optional: You can define a new PVC without having to specify the storage class:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: cinder-claim
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   A PVC that does not specify a specific storage class is automatically provisioned by using the default storage class.

6. Optional: After the new file has been configured, create it in your cluster:

   $ oc create -f cinder-claim.yaml

1. In the OpenShift Container Platform console, click Storage Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the required options on the resulting page.

   1. Select the appropriate storage class.
   2. Enter a unique name for the storage claim.

   3. Select the access mode to specify read and write access for the PVC you are creating.

      Important

      Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

4. Define the size of the storage claim.
5. Click Create to create the persistent volume claim and generate a persistent volume.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-manila.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-manila
   spec:
     accessModes: 1
       - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     storageClassName: csi-manila-gold 2

   1

   Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

   2

   The name of the storage class that provisions the storage back end. Manila storage classes are provisioned by the Operator and have the csi-manila- prefix.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-manila.yaml

   A new PVC is created.

3. To verify that the volume was created and is ready, run the following command:

   $ oc get pvc pvc-manila

   The pvc-manila shows that it is Bound.

1. Install the Secrets Store CSI Driver Operator:

   1. Log in to the web console.
   2. Click Operators OperatorHub.
   3. Locate the Secrets Store CSI Driver Operator by typing "Secrets Store CSI" in the filter box.
   4. Click the Secrets Store CSI Driver Operator button.
   5. On the Secrets Store CSI Driver Operator page, click Install.

   6. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   7. Click Install.

      After the installation finishes, the Secrets Store CSI Driver Operator is listed in the Installed Operators section of the web console.

2. Create the ClusterCSIDriver instance for the driver (secrets-store.csi.k8s.io):

   1. Click Administration CustomResourceDefinitions ClusterCSIDriver.

   2. On the Instances tab, click Create ClusterCSIDriver.

      Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: secrets-store.csi.k8s.io
      spec:
        managementState: Managed

   3. Click Create.

1. Stop all application pods that use the secrets-store.csi.k8s.io provider.
2. Remove any third-party provider plug-in for your chosen secret store.

3. Remove the Container Storage Interface (CSI) driver and associated manifests:

   1. Click Administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, for secrets-store.csi.k8s.io, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.
4. Verify that the CSI driver pods are no longer running.

5. Uninstall the Secrets Store CSI Driver Operator:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click Operators Installed Operators.
   2. On the Installed Operators page, scroll or type "Secrets Store CSI" into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the Secrets Store CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

1. Delete the third-party vSphere CSI Driver (VMware vSphere Container Storage Plugin) Deployment and Daemonset objects.
2. Delete the configmap and secret objects that were installed previously with the third-party vSphere CSI Driver.

3. Delete the third-party vSphere CSI driver CSIDriver object:

   ~ $ oc delete CSIDriver csi.vsphere.vmware.com

   csidriver.storage.k8s.io "csi.vsphere.vmware.com" deleted