Storage

1. Delete the PV.

   $ oc delete pv <pv-name>

   The associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume, still exists after the PV is deleted.

2. Clean up the data on the associated storage asset.
3. Delete the associated storage asset. Alternately, to reuse the same storage asset, create a new PV with the storage asset definition.

1. List the persistent volumes in your cluster:

   $ oc get pv

   Example output

   NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
    pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
    pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
    pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim3    manual                     3s

2. Choose one of your persistent volumes and change its reclaim policy:

   $ oc patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

3. Verify that your chosen persistent volume has the right policy:

   $ oc get pv

   Example output

   NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
    pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
    pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
    pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Retain          Bound     default/claim3    manual                     3s

   In the preceding output, the volume bound to claim default/claim3 now has a Retain reclaim policy. The volume will not be automatically deleted when a user deletes claim default/claim3.

1. ReadWriteOnce (RWO) volumes cannot be mounted on multiple nodes. If a node fails, the system does not allow the attached RWO volume to be mounted on a new node because it is already assigned to the failed node. If you encounter a multi-attach error message as a result, force delete the pod on a shutdown or crashed node to avoid data loss in critical workloads, such as when dynamic persistent volumes are attached.
2. Use a recreate deployment strategy for pods that rely on AWS EBS.
3. If the underlying vSphere environment supports the vSAN file service, then the vSphere Container Storage Interface (CSI) Driver Operator installed by OpenShift Container Platform supports provisioning of ReadWriteMany (RWX) volumes. If you do not have vSAN file service configured, and you request RWX, the volume fails to get created and an error is logged. For more information, see "Using Container Storage Interface" → "VMware vSphere CSI Driver Operator".

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the storage class created previously from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This determines the read and write access for the created storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Create a storage class:

   $ cat << EOF | oc create -f -
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <storage-class-name> 1
   parameters:
     fsType: ext4 2
     encrypted: "true"
     kmsKeyId: keyvalue 3
   provisioner: ebs.csi.aws.com
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   EOF

   1

   Specifies the name of the storage class.

   2

   File system that is created on provisioned volumes.

   3

   Specifies the full Amazon Resource Name (ARN) of the key to use when encrypting the container-persistent volume. If you do not provide any key, but the encrypted field is set to true, then the default KMS key is used. See Finding the key ID and key ARN on AWS in the AWS documentation.

2. Create a persistent volume claim (PVC) with the storage class specifying the KMS key:

   $ cat << EOF | oc create -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: mypvc
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Filesystem
     storageClassName: <storage-class-name>
     resources:
       requests:
         storage: 1Gi
   EOF

3. Create workload containers to consume the PVC:

   $ cat << EOF | oc create -f -
   kind: Pod
   metadata:
     name: mypod
   spec:
     containers:
       - name: httpd
         image: quay.io/centos7/httpd-24-centos7
         ports:
           - containerPort: 80
         volumeMounts:
           - mountPath: /mnt/storage
             name: data
     volumes:
       - name: data
         persistentVolumeClaim:
           claimName: mypvc
   EOF

Procedure

1. In the OpenShift Container Platform console, click Storage → Storage Classes.
2. In the storage class overview, click Create Storage Class.

3. Define the desired options on the page that appears.

   1. Enter a name to reference the storage class.
   2. Enter an optional description.
   3. Select the reclaim policy.

   4. Select kubernetes.io/azure-disk from the drop down list.

      1. Enter the storage account type. This corresponds to your Azure storage account SKU tier. Valid options are Premium_LRS, Standard_LRS, StandardSSD_LRS, and UltraSSD_LRS.

      2. Enter the kind of account. Valid options are shared, dedicated, and managed.

         Important

         Red Hat only supports the use of kind: Managed in the storage class.

         With Shared and Dedicated, Azure creates unmanaged disks, while OpenShift Container Platform creates a managed disk for machine OS (root) disks. But because Azure Disk does not allow the use of both managed and unmanaged disks on a node, unmanaged disks created with Shared or Dedicated cannot be attached to OpenShift Container Platform nodes.

   5. Enter additional parameters for the storage class as desired.
4. Click Create to create the storage class.

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the storage class created previously from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This determines the read and write access for the created storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Create a Secret object that contains the Azure File credentials:

   $ oc create secret generic <secret-name> --from-literal=azurestorageaccountname=<storage-account> \ 1
     --from-literal=azurestorageaccountkey=<storage-account-key> 2

   1

   The Azure File storage account name.

   2

   The Azure File storage account key.

2. Create a PersistentVolume object that references the Secret object you created:

   apiVersion: "v1"
   kind: "PersistentVolume"
   metadata:
     name: "pv0001" 1
   spec:
     capacity:
       storage: "5Gi" 2
     accessModes:
       - "ReadWriteOnce"
     storageClassName: azure-file-sc
     azureFile:
       secretName: <secret-name> 3
       shareName: share-1 4
       readOnly: false

   1

   The name of the persistent volume.

   2

   The size of this persistent volume.

   3

   The name of the secret that contains the Azure File share credentials.

   4

   The name of the Azure File share.

3. Create a PersistentVolumeClaim object that maps to the persistent volume you created:

   apiVersion: "v1"
   kind: "PersistentVolumeClaim"
   metadata:
     name: "claim1" 1
   spec:
     accessModes:
       - "ReadWriteOnce"
     resources:
       requests:
         storage: "5Gi" 2
     storageClassName: azure-file-sc 3
     volumeName: "pv0001" 4

   1

   The name of the persistent volume claim.

   2

   The size of this persistent volume claim.

   3

   The name of the storage class that is used to provision the persistent volume. Specify the storage class used in the PersistentVolume definition.

   4

   The name of the existing PersistentVolume object that references the Azure File share.

1. Ensure that the executable file exists on all nodes in the cluster.
2. Place the executable file at the volume plugin path: /etc/kubernetes/kubelet-plugins/volume/exec/<vendor>~<driver>/<driver>.

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the storage class created previously from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This determines the read and write access for the created storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Define the persistent volume (PV). Create a file, pv.yaml, with the PersistentVolume object definition:

     apiVersion: v1
     kind: PersistentVolume
     metadata:
       name: task-pv-volume 1
       labels:
         type: local
     spec:
       storageClassName: manual 2
       capacity:
         storage: 5Gi
       accessModes:
         - ReadWriteOnce 3
       persistentVolumeReclaimPolicy: Retain
       hostPath:
         path: "/mnt/data" 4

   1

   The name of the volume. This name is how it is identified by persistent volume claims or pods.

   2

   Used to bind persistent volume claim requests to this persistent volume.

   3

   The volume can be mounted as read-write by a single node.

   4

   The configuration file specifies that the volume is at /mnt/data on the cluster’s node. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system. It is safe to mount the host by using /host.

2. Create the PV from the file:

   $ oc create -f pv.yaml

3. Define the persistent volume claim (PVC). Create a file, pvc.yaml, with the PersistentVolumeClaim object definition:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: task-pvc-volume
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
     storageClassName: manual

4. Create the PVC from the file:

   $ oc create -f pvc.yaml

Procedure

1. Create the openshift-local-storage project:

   $ oc adm new-project openshift-local-storage

2. Optional: Allow local storage creation on infrastructure nodes.

   You might want to use the Local Storage Operator to create volumes on infrastructure nodes in support of components such as logging and monitoring.

   You must adjust the default node selector so that the Local Storage Operator includes the infrastructure nodes, and not just worker nodes.

   To block the Local Storage Operator from inheriting the cluster-wide default selector, enter the following command:

   $ oc annotate namespace openshift-local-storage openshift.io/node-selector=''

3. Optional: Allow local storage to run on the management pool of CPUs in single-node deployment.

   Use the Local Storage Operator in single-node deployments and allow the use of CPUs that belong to the managment pool. Perform this step on single-node installations that use management workload partitioning.

   To allow Local Storage Operator to run on the management CPU pool, run following commands:

   $ oc annotate namespace openshift-local-storage workload.openshift.io/allowed='management'

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → OperatorHub.
3. Type Local Storage into the filter box to locate the Local Storage Operator.
4. Click Install.
5. On the Install Operator page, select A specific namespace on the cluster. Select openshift-local-storage from the drop-down menu.
6. Adjust the values for Update Channel and Approval Strategy to the values that you want.
7. Click Install.

From the CLI

1. Install the Local Storage Operator from the CLI.

   1. Run the following command to get the OpenShift Container Platform major and minor version. It is required for the channel value in the next step.

      $ OC_VERSION=$(oc version -o yaml | grep openshiftVersion | \
          grep -o '[0-9]*[.][0-9]*' | head -1)

   2. Create an object YAML file to define an Operator group and subscription for the Local Storage Operator, such as openshift-local-storage.yaml:

      Example openshift-local-storage.yaml

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: local-operator-group
        namespace: openshift-local-storage
      spec:
        targetNamespaces:
          - openshift-local-storage
      ---
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: local-storage-operator
        namespace: openshift-local-storage
      spec:
        channel: "${OC_VERSION}"
        installPlanApproval: Automatic 1
        name: local-storage-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

      1

      The user approval policy for an install plan.

2. Create the Local Storage Operator object by entering the following command:

   $ oc apply -f openshift-local-storage.yaml

   At this point, the Operator Lifecycle Manager (OLM) is now aware of the Local Storage Operator. A ClusterServiceVersion (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

3. Verify local storage installation by checking that all pods and the Local Storage Operator have been created:

   1. Check that all the required pods have been created:

      $ oc -n openshift-local-storage get pods

      Example output

      NAME                                      READY   STATUS    RESTARTS   AGE
      local-storage-operator-746bf599c9-vlt5t   1/1     Running   0          19m

   2. Check the ClusterServiceVersion (CSV) YAML manifest to see that the Local Storage Operator is available in the openshift-local-storage project:

      $ oc get csvs -n openshift-local-storage

      Example output

      NAME                                         DISPLAY         VERSION               REPLACES   PHASE
      local-storage-operator.4.2.26-202003230335   Local Storage   4.2.26-202003230335              Succeeded

Procedure

1. Create the local volume resource. This resource must define the nodes and paths to the local volumes.

   Note

   Do not use different storage class names for the same device. Doing so will create multiple persistent volumes (PVs).

   Example: Filesystem

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 1
   spec:
     nodeSelector: 2
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-140-183
             - ip-10-0-158-139
             - ip-10-0-164-33
     storageClassDevices:
       - storageClassName: "local-sc" 3
         volumeMode: Filesystem 4
         fsType: xfs 5
         devicePaths: 6
           - /path/to/device 7

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   5

   The file system that is created when the local volume is mounted for the first time.

   6

   The path containing a list of local storage devices to choose from.

   7

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as /dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   Note

   A raw block volume (volumeMode: block) is not formatted with a file system. You should use this mode only if any application running on the pod can use raw block devices.

   Example: Block

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 1
   spec:
     nodeSelector: 2
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-136-143
             - ip-10-0-140-255
             - ip-10-0-144-180
     storageClassDevices:
       - storageClassName: "localblock-sc" 3
         volumeMode: Block  4
         devicePaths: 5
           - /path/to/device 6

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   5

   The path containing a list of local storage devices to choose from.

   6

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

2. Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <local-volume>.yaml

3. Verify that the provisioner was created and that the corresponding daemon sets were created:

   $ oc get all -n openshift-local-storage

   Example output

   NAME                                          READY   STATUS    RESTARTS   AGE
   pod/diskmaker-manager-9wzms                   1/1     Running   0          5m43s
   pod/diskmaker-manager-jgvjp                   1/1     Running   0          5m43s
   pod/diskmaker-manager-tbdsj                   1/1     Running   0          5m43s
   pod/local-storage-operator-7db4bd9f79-t6k87   1/1     Running   0          14m
   
   NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
   service/local-storage-operator-metrics   ClusterIP   172.30.135.36   <none>        8383/TCP,8686/TCP   14m
   
   NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/diskmaker-manager   3         3         3       3            3           <none>          5m43s
   
   NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/local-storage-operator   1/1     1            1           14m
   
   NAME                                                DESIRED   CURRENT   READY   AGE
   replicaset.apps/local-storage-operator-7db4bd9f79   1         1         1       14m

   Note the desired and current number of daemon set processes. A desired count of 0 indicates that the label selectors were invalid.

4. Verify that the persistent volumes were created:

   $ oc get pv

   Example output

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-1cec77cf   100Gi      RWO            Delete           Available           local-sc                88m
   local-pv-2ef7cd2a   100Gi      RWO            Delete           Available           local-sc                82m
   local-pv-3fa1c73    100Gi      RWO            Delete           Available           local-sc                48m

Procedure

1. Define the PV. Create a file, such as example-pv-filesystem.yaml or example-pv-block.yaml, with the PersistentVolume object definition. This resource must define the nodes and paths to the local volumes.

   Note

   Do not use different storage class names for the same device. Doing so will create multiple PVs.

   example-pv-filesystem.yaml

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: example-pv-filesystem
   spec:
     capacity:
       storage: 100Gi
     volumeMode: Filesystem 1
     accessModes:
     - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     storageClassName: local-storage 2
     local:
       path: /dev/xvdf 3
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - example-node

   1

   The volume mode, either Filesystem or Block, that defines the type of PVs.

   2

   The name of the storage class to use when creating PV resources. Use a storage class that uniquely identifies this set of PVs.

   3

   The path containing a list of local storage devices to choose from, or a directory. You can only specify a directory with Filesystem volumeMode.

   Note

   A raw block volume (volumeMode: block) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices.

   example-pv-block.yaml

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: example-pv-block
   spec:
     capacity:
       storage: 100Gi
     volumeMode: Block 1
     accessModes:
     - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     storageClassName: local-storage 2
     local:
       path: /dev/xvdf 3
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - example-node

   1

   The volume mode, either Filesystem or Block, that defines the type of PVs.

   2

   The name of the storage class to use when creating PV resources. Be sure to use a storage class that uniquely identifies this set of PVs.

   3

   The path containing a list of local storage devices to choose from.

2. Create the PV resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <example-pv>.yaml

3. Verify that the local PV was created:

   $ oc get pv

   Example output

   NAME                    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                STORAGECLASS    REASON   AGE
   example-pv-filesystem   100Gi      RWO            Delete           Available                        local-storage            3m47s
   example-pv1             1Gi        RWO            Delete           Bound       local-storage/pvc1   local-storage            12h
   example-pv2             1Gi        RWO            Delete           Bound       local-storage/pvc2   local-storage            12h
   example-pv3             1Gi        RWO            Delete           Bound       local-storage/pvc3   local-storage            12h

Procedure

1. Create the PVC using the corresponding storage class:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: local-pvc-name 1
   spec:
     accessModes:
     - ReadWriteOnce
     volumeMode: Filesystem 2
     resources:
       requests:
         storage: 100Gi 3
     storageClassName: local-sc 4

   1

   Name of the PVC.

   2

   The type of the PVC. Defaults to Filesystem.

   3

   The amount of storage available to the PVC.

   4

   Name of the storage class required by the claim.

2. Create the PVC in the OpenShift Container Platform cluster, specifying the file you just created:

   $ oc create -f <local-pvc>.yaml

Procedure

1. Include the defined claim in the resource spec. The following example declares the persistent volume claim inside a pod:

   apiVersion: v1
   kind: Pod
   spec:
     ...
     containers:
       volumeMounts:
       - name: local-disks 1
         mountPath: /data 2
     volumes:
     - name: localpvc
       persistentVolumeClaim:
         claimName: local-pvc-name 3

   1

   The name of the volume to mount.

   2

   The path inside the pod where the volume is mounted. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host /dev/pts files. It is safe to mount the host by using /host.

   3

   The name of the existing persistent volume claim to use.

2. Create the resource in the OpenShift Container Platform cluster, specifying the file you just created:

   $ oc create -f <local-pod>.yaml

Procedure

1. To enable automatic discovery of local devices from the web console:

   1. In the Administrator perspective, navigate to Operators → Installed Operators and click on the Local Volume Discovery tab.
   2. Click Create Local Volume Discovery.

   3. Select either All nodes or Select nodes, depending on whether you want to discover available disks on all or specific nodes.

      Note

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   4. Click Create.

1. To display a continuous list of available devices on a node:

   1. Log in to the OpenShift Container Platform web console.
   2. Navigate to Compute → Nodes.
   3. Click the node name that you want to open. The "Node Details" page is displayed.

   4. Select the Disks tab to display the list of the selected devices.

      The device list updates continuously as local disks are added or removed. You can filter the devices by name, status, type, model, capacity, and mode.

2. To automatically provision local volumes for the discovered devices from the web console:

   1. Navigate to Operators → Installed Operators and select Local Storage from the list of Operators.
   2. Select Local Volume Set → Create Local Volume Set.
   3. Enter a volume set name and a storage class name.

   4. Choose All nodes or Select nodes to apply filters accordingly.

      Note

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   5. Select the disk type, mode, size, and limit you want to apply to the local volume set, and click Create.

      A message displays after several minutes, indicating that the "Operator reconciled successfully."

3. Alternatively, to provision local volumes for the discovered devices from the CLI:

   1. Create an object YAML file to define the local volume set, such as local-volume-set.yaml, as shown in the following example:

      apiVersion: local.storage.openshift.io/v1alpha1
      kind: LocalVolumeSet
      metadata:
        name: example-autodetect
      spec:
        nodeSelector:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                    - worker-0
                    - worker-1
        storageClassName: example-storageclass 1
        volumeMode: Filesystem
        fsType: ext4
        maxDeviceCount: 10
        deviceInclusionSpec:
          deviceTypes: 2
            - disk
            - part
          deviceMechanicalProperties:
            - NonRotational
          minSize: 10G
          maxSize: 100G
          models:
            - SAMSUNG
            - Crucial_CT525MX3
          vendors:
            - ATA
            - ST2000LM

      1

      Determines the storage class that is created for persistent volumes that are provisioned from discovered devices. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.

      2

      When using the local volume set feature, the Local Storage Operator does not support the use of logical volume management (LVM) devices.

   2. Create the local volume set object:

      $ oc apply -f local-volume-set.yaml

   3. Verify that the local persistent volumes were dynamically provisioned based on the storage class:

      $ oc get pv

      Example output

      NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS           REASON   AGE
      local-pv-1cec77cf   100Gi      RWO            Delete           Available           example-storageclass            88m
      local-pv-2ef7cd2a   100Gi      RWO            Delete           Available           example-storageclass            82m
      local-pv-3fa1c73    100Gi      RWO            Delete           Available           example-storageclass            48m

1. Modify the YAML file that defines the Pod and add the LocalVolume spec, as shown in the following example:

     apiVersion: "local.storage.openshift.io/v1"
     kind: "LocalVolume"
     metadata:
       name: "local-disks"
       namespace: "openshift-local-storage"
     spec:
       tolerations:
         - key: localstorage 1
           operator: Equal 2
           value: "localstorage" 3
       storageClassDevices:
           - storageClassName: "localblock-sc"
             volumeMode: Block 4
             devicePaths: 5
               - /dev/xvdg

   1

   Specify the key that you added to the node.

   2

   Specify the Equal operator to require the key/value parameters to match. If operator is Exists, the system checks that the key exists and ignores the value. If operator is Equal, then the key and value must match.

   3

   Specify the value local of the tainted node.

   4

   The volume mode, either Filesystem or Block, defining the type of the local volumes.

   5

   The path containing a list of local storage devices to choose from.

2. Optional: To create local persistent volumes on only tainted nodes, modify the YAML file and add the LocalVolume spec, as shown in the following example:

   spec:
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists

Procedure

1. Create an object definition for the PV:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001 1
   spec:
     capacity:
       storage: 5Gi 2
     accessModes:
     - ReadWriteOnce 3
     nfs: 4
       path: /tmp 5
       server: 172.17.0.2 6
     persistentVolumeReclaimPolicy: Retain 7

   1

   The name of the volume. This is the PV identity in various oc <command> pod commands.

   2

   The amount of storage allocated to this volume.

   3

   Though this appears to be related to controlling access to the volume, it is actually used similarly to labels and used to match a PVC to a PV. Currently, no access rules are enforced based on the accessModes.

   4

   The volume type being used, in this case the nfs plugin.

   5

   The path that is exported by the NFS server.

   6

   The hostname or IP address of the NFS server.

   7

   The reclaim policy for the PV. This defines what happens to a volume when released.

   Note

   Each NFS volume must be mountable by all schedulable nodes in the cluster.

2. Verify that the PV was created:

   $ oc get pv

   Example output

   NAME     LABELS    CAPACITY     ACCESSMODES   STATUS      CLAIM  REASON    AGE
   pv0001   <none>    5Gi          RWO           Available                    31s

3. Create a persistent volume claim that binds to the new PV:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: nfs-claim1
   spec:
     accessModes:
       - ReadWriteOnce 1
     resources:
       requests:
         storage: 5Gi 2
     volumeName: pv0001
     storageClassName: ""

   1

   The access modes do not enforce security, but rather act as labels to match a PV to a PVC.

   2

   This claim looks for PVs offering 5Gi or greater capacity.

4. Verify that the persistent volume claim was created:

   $ oc get pvc

   Example output

   NAME         STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   nfs-claim1   Bound    pv0001   5Gi        RWO                           2m

Procedure

1. Create the virtual machine disks. Virtual machine disks (VMDKs) must be created manually before statically provisioning VMware vSphere volumes. Use either of the following methods:

   • Create using vmkfstools. Access ESX through Secure Shell (SSH) and then use following command to create a VMDK volume:

     $ vmkfstools -c <size> /vmfs/volumes/<datastore-name>/volumes/<disk-name>.vmdk

   • Create using vmware-diskmanager:

     $ shell vmware-vdiskmanager -c -t 0 -s <size> -a lsilogic <disk-name>.vmdk

2. Create a persistent volume that references the VMDKs. Create a file, pv1.yaml, with the PersistentVolume object definition:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv1 1
   spec:
     capacity:
       storage: 1Gi 2
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain
     vsphereVolume: 3
       volumePath: "[datastore1] volumes/myDisk"  4
       fsType: ext4  5

   1

   The name of the volume. This name is how it is identified by persistent volume claims or pods.

   2

   The amount of storage allocated to this volume.

   3

   The volume type used, with vsphereVolume for vSphere volumes. The label is used to mount a vSphere VMDK volume into pods. The contents of a volume are preserved when it is unmounted. The volume type supports VMFS and VSAN datastore.

   4

   The existing VMDK volume to use. If you used vmkfstools, you must enclose the datastore name in square brackets, [], in the volume definition, as shown previously.

   5

   The file system type to mount. For example, ext4, xfs, or other file systems.

   Important

   Changing the value of the fsType parameter after the volume is formatted and provisioned can result in data loss and pod failure.

3. Create the PersistentVolume object from the file:

   $ oc create -f pv1.yaml

4. Create a persistent volume claim that maps to the persistent volume you created in the previous step. Create a file, pvc1.yaml, with the PersistentVolumeClaim object definition:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc1 1
   spec:
     accessModes:
       - ReadWriteOnce 2
     resources:
      requests:
        storage: "1Gi" 3
     volumeName: pv1 4

   1

   A unique name that represents the persistent volume claim.

   2

   The access mode of the persistent volume claim. With ReadWriteOnce, the volume can be mounted with read and write permissions by a single node.

   3

   The size of the persistent volume claim.

   4

   The name of the existing persistent volume.

5. Create the PersistentVolumeClaim object from the file:

   $ oc create -f pvc1.yaml

Procedure

1. Create the Pod object definition and save it to a file.

2. Embed the CSI inline ephemeral volume in the file.

   my-csi-app.yaml

   kind: Pod
   apiVersion: v1
   metadata:
     name: my-csi-app
   spec:
     containers:
       - name: my-frontend
         image: busybox
         volumeMounts:
         - mountPath: "/data"
           name: my-csi-inline-vol
         command: [ "sleep", "1000000" ]
     volumes: 1
       - name: my-csi-inline-vol
         csi:
           driver: inline.storage.kubernetes.io
           volumeAttributes:
             foo: bar

   1

   The name of the volume that is used by pods.

3. Create the object definition file that you saved in the previous step.

   $ oc create -f my-csi-app.yaml

Procedure

1. Grant a given service account RBAC permissions to use the SharedSecret CR instance in its pod by using oc apply with YAML content:

   Note

   Currently, kubectl and oc have hard-coded special case logic restricting the use verb to roles centered around pod security. Therefore, you cannot use oc create role …​ to create the role needed for consuming SharedSecret CR instances.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedsecrets
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the RoleBinding associated with the role by using the oc command:

   $ oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the SharedSecret CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedSecret: my-share
   
   EOF

Procedure

1. Create a SharedConfigMap CR instance for the config map that you want to share across namespaces in the cluster:

   $ oc apply -f - <<EOF
   apiVersion: sharedresource.openshift.io/v1alpha1
   kind: SharedConfigMap
   metadata:
     name: my-share
   spec:
     configMapRef:
       name: <name of configmap>
       namespace: <namespace of configmap>
   EOF

Procedure

1. Grant a given service account RBAC permissions to use the SharedConfigMap CR instance in its pod by using oc apply with YAML content.

   Note

   Currently, kubectl and oc have hard-coded special case logic restricting the use verb to roles centered around pod security. Therefore, you cannot use oc create role …​ to create the role needed for consuming a SharedConfigMap CR instance.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedconfigmaps
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the RoleBinding associated with the role by using the oc command:

   oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the SharedConfigMap CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedConfigMap: my-share
   
   EOF

1. Create a file with the VolumeSnapshotClass object described by the following YAML:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io 1
   deletionPolicy: Delete

   1

   The name of the CSI driver that is used to create snapshots of this VolumeSnapshotClass object. The name must be the same as the Provisioner field of the storage class that is responsible for the PVC that is being snapshotted.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshotclass.yaml

3. Create a VolumeSnapshot object:

   volumesnapshot-dynamic.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     volumeSnapshotClassName: csi-hostpath-snap 1
     source:
       persistentVolumeClaimName: myclaim 2

   1

   The request for a particular class by the volume snapshot. If the volumeSnapshotClassName setting is absent and there is a default volume snapshot class, a snapshot is created with the default volume snapshot class name. But if the field is absent and no default volume snapshot class exists, then no snapshot is created.

   2

   The name of the PersistentVolumeClaim object bound to a persistent volume. This defines what you want to create a snapshot of. Required for dynamically provisioning a snapshot.

4. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-dynamic.yaml

1. Provide a value for the volumeSnapshotContentName parameter as the source for the snapshot, in addition to defining volume snapshot class as shown above.

   volumesnapshot-manual.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: snapshot-demo
   spec:
     source:
       volumeSnapshotContentName: mycontent 1

   1

   The volumeSnapshotContentName parameter is required for pre-provisioned snapshots.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-manual.yaml

1. To display details about the volume snapshot that was created, enter the following command:

   $ oc describe volumesnapshot mysnap

   The following example displays details about the mysnap volume snapshot:

   volumesnapshot.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     source:
       persistentVolumeClaimName: myclaim
     volumeSnapshotClassName: csi-hostpath-snap
   status:
     boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 1
     creationTime: "2020-01-29T12:24:30Z" 2
     readyToUse: true 3
     restoreSize: 500Mi

   1

   The pointer to the actual storage content that was created by the controller.

   2

   The time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time.

   3

   If the value is set to true, the snapshot can be used to restore as a new PVC.
   If the value is set to false, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes.

2. To verify that the volume snapshot was created, enter the following command:

   $ oc get volumesnapshotcontent

   The pointer to the actual content is displayed. If the boundVolumeSnapshotContentName field is populated, a VolumeSnapshotContent object exists and the snapshot was created.

3. To verify that the snapshot is ready, confirm that the VolumeSnapshot object has readyToUse: true.

Procedure

1. Specify the deletion policy that you require in the VolumeSnapshotClass object, as shown in the following example:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io
   deletionPolicy: Delete 1

   1

   When deleting the volume snapshot, if the Delete value is set, the underlying snapshot is deleted along with the VolumeSnapshotContent object. If the Retain value is set, both the underlying snapshot and VolumeSnapshotContent object remain.
   If the Retain value is set and the VolumeSnapshot object is deleted without deleting the corresponding VolumeSnapshotContent object, the content remains. The snapshot itself is also retained in the storage back end.

2. Delete the volume snapshot by entering the following command:

   $ oc delete volumesnapshot <volumesnapshot_name>

   Example output

   volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted

3. If the deletion policy is set to Retain, delete the volume snapshot content by entering the following command:

   $ oc delete volumesnapshotcontent <volumesnapshotcontent_name>

4. Optional: If the VolumeSnapshot object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue:

   Important

   Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the VolumeSnapshot object. Even with the --force option, the delete operation does not delete snapshot objects until all finalizers are removed.

   $ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}'

   Example output

   volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted

   The finalizers are removed and the volume snapshot is deleted.

Procedure

1. Specify a VolumeSnapshot data source on a PVC as shown in the following:

   pvc-restore.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: myclaim-restore
   spec:
     storageClassName: csi-hostpath-sc
     dataSource:
       name: mysnap 1
       kind: VolumeSnapshot 2
       apiGroup: snapshot.storage.k8s.io 3
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   1

   Name of the VolumeSnapshot object representing the snapshot to use as source.

   2

   Must be set to the VolumeSnapshot value.

   3

   Must be set to the snapshot.storage.k8s.io value.

2. Create a PVC by entering the following command:

   $ oc create -f pvc-restore.yaml

3. Verify that the restored PVC has been created by entering the following command:

   $ oc get pvc

   A new PVC such as myclaim-restore is displayed.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-clone.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-1-clone
     namespace: mynamespace
   spec:
     storageClassName: csi-cloning 1
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
     dataSource:
       kind: PersistentVolumeClaim
       name: pvc-1

   1

   The name of the storage class that provisions the storage back end. The default storage class can be used and storageClassName can be omitted in the spec.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-clone.yaml

   A new PVC pvc-1-clone is created.

3. Verify that the volume clone was created and is ready by running the following command:

   $ oc get pvc pvc-1-clone

   The pvc-1-clone shows that it is Bound.

   You are now ready to use the newly cloned PVC to configure a pod.

4. Create and save a file with the Pod object described by the YAML. For example:

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
       - name: myfrontend
         image: dockerfile/nginx
         volumeMounts:
         - mountPath: "/var/www/html"
           name: mypd
     volumes:
       - name: mypd
         persistentVolumeClaim:
           claimName: pvc-1-clone 1

   1

   The cloned PVC created during the CSI volume cloning operation.

   The created Pod object is now ready to consume, clone, snapshot, or delete your cloned PVC independently of its original dataSource PVC.

1. Log in to the web console.

2. Install the AWS EFS CSI Operator:

   1. Click Operators → OperatorHub.
   2. Locate the AWS EFS CSI Operator by typing AWS EFS CSI in the filter box.

   3. Click the AWS EFS CSI Driver Operator button.

      Important

      Be sure to select the AWS EFS CSI Driver Operator and not the AWS EFS Operator. The AWS EFS Operator is a community Operator and is not supported by Red Hat.

   4. On the AWS EFS CSI Driver Operator page, click Install.

   5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   6. Click Install.

      After the installation finishes, the AWS EFS CSI Operator is listed in the Installed Operators section of the web console.

3. If you are using AWS EFS with AWS Security Token Service (STS), you must configure the AWS EFS CSI Driver with STS. For more information, see "Configuring AWS EFS CSI Driver with STS".

4. Install the AWS EFS CSI Driver:

   1. Click administration → CustomResourceDefinitions → ClusterCSIDriver.
   2. On the Instances tab, click Create ClusterCSIDriver.

   3. Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: efs.csi.aws.com
      spec:
        managementState: Managed

   4. Click Create.

   5. Wait for the following Conditions to change to a "true" status:

      • AWSEFSDriverCredentialsRequestControllerAvailable
      • AWSEFSDriverNodeServiceControllerAvailable
      • AWSEFSDriverControllerServiceControllerAvailable

1. Extract the CCO utility (ccoctl) binary from the OpenShift Container Platform release image, which you used to install the cluster with STS. For more information, see "Configuring the Cloud Credential Operator utility".

2. Create and save an EFS CredentialsRequest YAML file, such as shown in the following example, and then place it in the credrequests directory:

   Example

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: openshift-aws-efs-csi-driver
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - action:
         - elasticfilesystem:*
         effect: Allow
         resource: '*'
     secretRef:
       name: aws-efs-cloud-credentials
       namespace: openshift-cluster-csi-drivers
     serviceAccountNames:
     - aws-efs-csi-driver-operator
     - aws-efs-csi-driver-controller-sa

3. Run the ccoctl tool to generate a new IAM role in AWS, and create a YAML file for it in the local file system (<path_to_ccoctl_output_dir>/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml).

   $ ccoctl aws create-iam-roles --name=<name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

   • name=<name> is the name used to tag any cloud resources that are created for tracking.
   • region=<aws_region> is the AWS region where cloud resources are created.
   • dir=<path_to_directory_with_list_of_credentials_requests>/credrequests is the directory containing the EFS CredentialsRequest file in previous step.

   • <aws_account_id> is the AWS account ID.

     Example

     $ ccoctl aws create-iam-roles --name my-aws-efs --credentials-requests-dir credrequests --identity-provider-arn arn:aws:iam::123456789012:oidc-provider/my-aws-efs-oidc.s3.us-east-2.amazonaws.com

     Example output

     2022/03/21 06:24:44 Role arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud- created
     2022/03/21 06:24:44 Saved credentials configuration to: /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml
     2022/03/21 06:24:45 Updated Role policy for Role my-aws-efs-openshift-cluster-csi-drivers-aws-efs-cloud-

4. Create the AWS EFS cloud credentials and secret:

   $ oc create -f <path_to_ccoctl_output_dir>/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml

   Example

   $ oc create -f /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml

   Example output

   secret/aws-efs-cloud-credentials created

1. On the AWS console, open https://console.aws.amazon.com/efs.

2. Click Create file system:

   • Enter a name for the file system.
   • For Virtual Private Cloud (VPC), select your OpenShift Container Platform’s' virtual private cloud (VPC).
   • Accept default settings for all other selections.

3. Wait for the volume and mount targets to finish being fully created:

   1. Go to https://console.aws.amazon.com/efs#/file-systems.
   2. Click your volume, and on the Network tab wait for all mount targets to become available (~1-2 minutes).
4. On the Network tab, copy the Security Group ID (you will need this in the next step).
5. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups, and find the Security Group used by the EFS volume.

6. On the Inbound rules tab, click Edit inbound rules, and then add a new rule with the following settings to allow OpenShift Container Platform nodes to access EFS volumes :

   • Type: NFS
   • Protocol: TCP
   • Port range: 2049

   • Source: Custom/IP address range of your nodes (for example: “10.0.0.0/16”)

     This step allows OpenShift Container Platform to use NFS ports from the cluster.

7. Save the rule.

1. Log in to the web console.
2. Stop all applications that use AWS EFS PVs.

3. Delete all AWS EFS PVs:

   1. Click Storage → PersistentVolumeClaims.
   2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

4. Uninstall the AWS EFS CSI Driver:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click administration → CustomResourceDefinitions → ClusterCSIDriver.
   2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.

5. Uninstall the AWS EFS CSI Operator:

   1. Click Operators → Installed Operators.
   2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions → Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

1. Create a storage class designating the storage account type using a YAML file similar to the following:

   $ oc create -f - << EOF
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <storage-class> 1
   provisioner: disk.csi.azure.com
   parameters:
     skuName: <storage-class-account-type> 2
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   EOF

   1

   Storage class name.

   2

   Storage account type. This corresponds to your Azure storage account SKU tier:`Standard_LRS`, Premium_LRS, StandardSSD_LRS, UltraSSD_LRS, Premium_ZRS, StandardSSD_ZRS.

2. Ensure that the storage class was created by listing the storage classes:

   $ oc get storageclass

   Example output

   $ oc get storageclass
   NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   68m
   managed-csi (default)   disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   68m
   sc-prem-zrs             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   4m25s 1

   1

   New storage class with storage account type.

Procedure

1. Enable feature gates with the TechPreviewNoUpgrade feature set (see Nodes → Enabling features using feature gates).

   Important

   After turning Technology Preview features on by using feature gates, they cannot be turned off and cluster upgrades are prevented.

2. Verify the cluster operator storage:

   $ oc get co storage

   NAME    VERSION                                 AVAILABLE   PROGRESSING DEGRADED   SINCE
   storage 4.10.0-0.nightly-2021-11-15-034648      True        False       False      4m36s

   • AVAILABLE should be "True".
   • PROGRESSING should be "False".
   • DEGRADED should be "False".

3. Verify the status of the pods in the openshift-cluster-csi-drivers namespace to ensure that they are running:

   $ oc get pod -n openshift-cluster-csi-drivers

   NAME                                                    READY   STATUS  RESTARTS    AGE
   azure-file-csi-driver-controller-5949bf45fd-pm4qb       11/11   Running 0           39m
   azure-file-csi-driver-node-2tcxr                        3/3     Running 0           53m
   azure-file-csi-driver-node-2xjzm                        3/3     Running 0           53m
   azure-file-csi-driver-node-6wrgk                        3/3     Running 0           53m
   azure-file-csi-driver-node-frvx2                        3/3     Running 0           53m
   azure-file-csi-driver-node-lf5kb                        3/3     Running 0           53m
   azure-file-csi-driver-node-mqdhh                        3/3     Running 0           53m
   azure-file-csi-driver-operator-7d966fc6c5-x74x5         1/1     Running 0           44m

4. Verify that the storage class is installed:

   $ oc get storageclass

   NAME                        PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE       ALLOWVOLUMEEXPANSION    AGE
   azurefile-csi               file.csi.azure.com              Delete          Immediate               true                    10m 1
   managed-csi                 disk.csi.azure.com              Delete          WaitForFirstConsumer    true                    35m
   managed-premium (default)   kubernetes.io/azure-disk        Delete          WaitForFirstConsumer    true                    35m

   1

   Azure File storage class

1. Create a storage class with the Cloud KMS key. The following example enables dynamic provisioning of encrypted volumes:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: csi-gce-pd-cmek
   provisioner: pd.csi.storage.gke.io
   volumeBindingMode: "WaitForFirstConsumer"
   allowVolumeExpansion: true
   parameters:
     type: pd-standard
     disk-encryption-kms-key: projects/<key-project-id>/locations/<location>/keyRings/<key-ring>/cryptoKeys/<key> 1

   1

   This field must be the resource identifier for the key that will be used to encrypt new disks. Values are case-sensitive. For more information about providing key ID values, see Retrieving a resource’s ID and Getting a Cloud KMS resource ID.

   Note

   You cannot add the disk-encryption-kms-key parameter to an existing storage class. However, you can delete the storage class and recreate it with the same name and a different set of parameters. If you do this, the provisioner of the existing class must be pd.csi.storage.gke.io.

2. Deploy the storage class on your OpenShift Container Platform cluster using the oc command:

   $ oc describe storageclass csi-gce-pd-cmek

   Example output

   Name:                  csi-gce-pd-cmek
   IsDefaultClass:        No
   Annotations:           None
   Provisioner:           pd.csi.storage.gke.io
   Parameters:            disk-encryption-kms-key=projects/key-project-id/locations/location/keyRings/ring-name/cryptoKeys/key-name,type=pd-standard
   AllowVolumeExpansion:  true
   MountOptions:          none
   ReclaimPolicy:         Delete
   VolumeBindingMode:     WaitForFirstConsumer
   Events:                none

3. Create a file named pvc.yaml that matches the name of your storage class object that you created in the previous step:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: podpvc
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: csi-gce-pd-cmek
     resources:
       requests:
         storage: 6Gi

   Note

   If you marked the new storage class as default, you can omit the storageClassName field.

4. Apply the PVC on your cluster:

   $ oc apply -f pvc.yaml

5. Get the status of your PVC and verify that it is created and bound to a newly provisioned PV:

   $ oc get pvc

   Example output

   NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
   podpvc    Bound     pvc-e36abf50-84f3-11e8-8538-42010a800002   10Gi       RWO            csi-gce-pd-cmek  9s

   Note

   If your storage class has the volumeBindingMode field set to WaitForFirstConsumer, you must create a pod to use the PVC before you can verify it.

1. List the storage class:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard(default)      cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h
   standard-csi           kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h

2. Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default storage class, as shown in the following example:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

3. Make another storage class the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.

   $ oc patch storageclass standard-csi -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

4. Verify that the PVC is now referencing the CSI storage class by default:

   $ oc get storageclass

   Example output

   NAME                   PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard               kubernetes.io/cinder       Delete          WaitForFirstConsumer   true                   46h
   standard-csi(default)  cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   46h

5. Optional: You can define a new PVC without having to specify the storage class:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: cinder-claim
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   A PVC that does not specify a specific storage class is automatically provisioned by using the default storage class.

6. Optional: After the new file has been configured, create it in your cluster:

   $ oc create -f cinder-claim.yaml

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the required options on the resulting page.

   1. Select the appropriate storage class.
   2. Enter a unique name for the storage claim.

   3. Select the access mode to specify read and write access for the PVC you are creating.

      Important

      Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

4. Define the size of the storage claim.
5. Click Create to create the persistent volume claim and generate a persistent volume.

1. Create and save a file with the PersistentVolumeClaim object described by the following YAML:

   pvc-manila.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-manila
   spec:
     accessModes: 1
       - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     storageClassName: csi-manila-gold 2

   1

   Use RWX if you want the persistent volume (PV) that fulfills this PVC to be mounted to multiple pods on multiple nodes in the cluster.

   2

   The name of the storage class that provisions the storage back end. Manila storage classes are provisioned by the Operator and have the csi-manila- prefix.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-manila.yaml

   A new PVC is created.

3. To verify that the volume was created and is ready, run the following command:

   $ oc get pvc pvc-manila

   The pvc-manila shows that it is Bound.

Procedure

1. Delete the third-party vSphere CSI Driver (VMware vSphere Container Storage Plugin) Deployment and Daemonset objects.
2. Delete the configmap and secret objects that were installed previously with the third-party vSphere CSI Driver.

3. Delete the third-party vSphere CSI driver CSIDriver object:

   ~ $ oc delete CSIDriver csi.vsphere.vmware.com

   csidriver.storage.k8s.io "csi.vsphere.vmware.com" deleted