Storage

1. Delete the PV by running the following command:

   $ oc delete pv <pv_name>

   The associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume, still exists after the PV is deleted.

2. Clean up the data on the associated storage asset.
3. Delete the associated storage asset. Alternately, to reuse the same storage asset, create a new PV with the storage asset definition.

Procedure

1. List the persistent volumes in your cluster:

   $ oc get pv

   Example output

   NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
    pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
    pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
    pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim3    manual                     3s

2. Choose one of your persistent volumes and change its reclaim policy:

   $ oc patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

3. Verify that your chosen persistent volume has the right policy:

   $ oc get pv

   Example output

   NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
    pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
    pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
    pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Retain          Bound     default/claim3    manual                     3s

   In the preceding output, the volume bound to claim

   default/claim3

   now has a

   Retain

   reclaim policy. The volume will not be automatically deleted when a user deletes claim

   default/claim3

   .

1. ReadWriteOncePod access mode for persistent volumes is a Technology Preview feature.

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the previously-created storage class from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This selection determines the read and write access for the storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Create a storage class:

   $ cat << EOF | oc create -f -
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <storage-class-name> 

   1

   
   parameters:
     fsType: ext4 

   2

   
     encrypted: "true"
     kmsKeyId: keyvalue 

   3

   
   provisioner: ebs.csi.aws.com
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   EOF

   1

   Specifies the name of the storage class.

   2

   File system that is created on provisioned volumes.

   3

   Specifies the full Amazon Resource Name (ARN) of the key to use when encrypting the container-persistent volume. If you do not provide any key, but the encrypted field is set to true, then the default KMS key is used. See Finding the key ID and key ARN on AWS in the AWS documentation.

2. Create a persistent volume claim (PVC) with the storage class specifying the KMS key:

   $ cat << EOF | oc create -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: mypvc
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Filesystem
     storageClassName: <storage-class-name>
     resources:
       requests:
         storage: 1Gi
   EOF

3. Create workload containers to consume the PVC:

   $ cat << EOF | oc create -f -
   kind: Pod
   metadata:
     name: mypod
   spec:
     containers:
       - name: httpd
         image: quay.io/centos7/httpd-24-centos7
         ports:
           - containerPort: 80
         volumeMounts:
           - mountPath: /mnt/storage
             name: data
     volumes:
       - name: data
         persistentVolumeClaim:
           claimName: mypvc
   EOF

Procedure

1. In the OpenShift Container Platform console, click Storage → Storage Classes.
2. In the storage class overview, click Create Storage Class.

3. Define the desired options on the page that appears.

   1. Enter a name to reference the storage class.
   2. Enter an optional description.
   3. Select the reclaim policy.

   4. Select

      kubernetes.io/azure-disk

      from the drop down list.
      1. Enter the storage account type. This corresponds to your Azure storage account SKU tier. Valid options are

         Premium_LRS

         ,

         Standard_LRS

         ,

         StandardSSD_LRS

         , and

         UltraSSD_LRS

         .

      2. Enter the kind of account. Valid options are

         shared

         ,

         dedicated,

         and

         managed

         .
         Important

         Red Hat only supports the use of

         kind: Managed

         in the storage class.

         With

         Shared

         and

         Dedicated

         , Azure creates unmanaged disks, while OpenShift Container Platform creates a managed disk for machine OS (root) disks. But because Azure Disk does not allow the use of both managed and unmanaged disks on a node, unmanaged disks created with

         Shared

         or

         Dedicated

         cannot be attached to OpenShift Container Platform nodes.

   5. Enter additional parameters for the storage class as desired.
4. Click Create to create the storage class.

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the previously-created storage class from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This selection determines the read and write access for the storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Create a

   Secret

   object that contains the Azure File credentials:

   $ oc create secret generic <secret-name> --from-literal=azurestorageaccountname=<storage-account> \ 

   1

   
     --from-literal=azurestorageaccountkey=<storage-account-key> 

   2

   1

   The Azure File storage account name.

   2

   The Azure File storage account key.

2. Create a

   PersistentVolume

   object that references the

   Secret

   object you created:

   apiVersion: "v1"
   kind: "PersistentVolume"
   metadata:
     name: "pv0001" 

   1

   
   spec:
     capacity:
       storage: "5Gi" 

   2

   
     accessModes:
       - "ReadWriteOnce"
     storageClassName: azure-file-sc
     azureFile:
       secretName: <secret-name> 

   3

   
       shareName: share-1 

   4

   
       readOnly: false

   1

   The name of the persistent volume.

   2

   The size of this persistent volume.

   3

   The name of the secret that contains the Azure File share credentials.

   4

   The name of the Azure File share.

3. Create a

   PersistentVolumeClaim

   object that maps to the persistent volume you created:

   apiVersion: "v1"
   kind: "PersistentVolumeClaim"
   metadata:
     name: "claim1" 

   1

   
   spec:
     accessModes:
       - "ReadWriteOnce"
     resources:
       requests:
         storage: "5Gi" 

   2

   
     storageClassName: azure-file-sc 

   3

   
     volumeName: "pv0001" 

   4

   1

   The name of the persistent volume claim.

   2

   The size of this persistent volume claim.

   3

   The name of the storage class that is used to provision the persistent volume. Specify the storage class used in the PersistentVolume definition.

   4

   The name of the existing PersistentVolume object that references the Azure File share.

1. Ensure that the executable file exists on all nodes in the cluster.
2. Place the executable file at the volume plugin path:

   /etc/kubernetes/kubelet-plugins/volume/exec/<vendor>~<driver>/<driver>

   .

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the desired options on the page that appears.

   1. Select the previously-created storage class from the drop-down menu.
   2. Enter a unique name for the storage claim.
   3. Select the access mode. This selection determines the read and write access for the storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure

1. Create an object definition for the PV:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001 

   1

   
   spec:
     capacity:
       storage: 5Gi 

   2

   
     accessModes:
     - ReadWriteOnce 

   3

   
     nfs: 

   4

   
       path: /tmp 

   5

   
       server: 172.17.0.2 

   6

   
     persistentVolumeReclaimPolicy: Retain 

   7

   1

   The name of the volume. This is the PV identity in various oc <command> pod commands.

   2

   The amount of storage allocated to this volume.

   3

   Though this appears to be related to controlling access to the volume, it is actually used similarly to labels and used to match a PVC to a PV. Currently, no access rules are enforced based on the accessModes.

   4

   The volume type being used, in this case the nfs plugin.

   5

   The path that is exported by the NFS server.

   6

   The hostname or IP address of the NFS server.

   7

   The reclaim policy for the PV. This defines what happens to a volume when released.

   Note

   Each NFS volume must be mountable by all schedulable nodes in the cluster.

2. Verify that the PV was created:

   $ oc get pv

   Example output

   NAME     LABELS    CAPACITY     ACCESSMODES   STATUS      CLAIM  REASON    AGE
   pv0001   <none>    5Gi          RWO           Available                    31s

3. Create a persistent volume claim that binds to the new PV:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: nfs-claim1
   spec:
     accessModes:
       - ReadWriteOnce 

   1

   
     resources:
       requests:
         storage: 5Gi 

   2

   
     volumeName: pv0001
     storageClassName: ""

   1

   The access modes do not enforce security, but rather act as labels to match a PV to a PVC.

   2

   This claim looks for PVs offering 5Gi or greater capacity.

4. Verify that the persistent volume claim was created:

   $ oc get pvc

   Example output

   NAME         STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   nfs-claim1   Bound    pv0001   5Gi        RWO                           2m

Procedure

1. Create the virtual machine disks. Virtual machine disks (VMDKs) must be created manually before statically provisioning VMware vSphere volumes. Use either of the following methods:

   • Create using

     vmkfstools

     . Access ESX through Secure Shell (SSH) and then use following command to create a VMDK volume:

     $ vmkfstools -c <size> /vmfs/volumes/<datastore-name>/volumes/<disk-name>.vmdk

   • Create using

     vmware-diskmanager

     :

     $ shell vmware-vdiskmanager -c -t 0 -s <size> -a lsilogic <disk-name>.vmdk

2. Create a persistent volume that references the VMDKs. Create a file,

   pv1.yaml

   , with the

   PersistentVolume

   object definition:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv1 

   1

   
   spec:
     capacity:
       storage: 1Gi 

   2

   
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain
     vsphereVolume: 

   3

   
       volumePath: "[datastore1] volumes/myDisk"  

   4

   
       fsType: ext4  

   5

   1

   The name of the volume. This name is how it is identified by persistent volume claims or pods.

   2

   The amount of storage allocated to this volume.

   3

   The volume type used, with vsphereVolume for vSphere volumes. The label is used to mount a vSphere VMDK volume into pods. The contents of a volume are preserved when it is unmounted. The volume type supports VMFS and VSAN datastore.

   4

   The existing VMDK volume to use. If you used vmkfstools, you must enclose the datastore name in square brackets, [], in the volume definition, as shown previously.

   5

   The file system type to mount. For example, ext4, xfs, or other file systems.

   Important

   Changing the value of the fsType parameter after the volume is formatted and provisioned can result in data loss and pod failure.

3. Create the

   PersistentVolume

   object from the file:

   $ oc create -f pv1.yaml

4. Create a persistent volume claim that maps to the persistent volume you created in the previous step. Create a file,

   pvc1.yaml

   , with the

   PersistentVolumeClaim

   object definition:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc1 

   1

   
   spec:
     accessModes:
       - ReadWriteOnce 

   2

   
     resources:
      requests:
        storage: "1Gi" 

   3

   
     volumeName: pv1 

   4

   1

   A unique name that represents the persistent volume claim.

   2

   The access mode of the persistent volume claim. With ReadWriteOnce, the volume can be mounted with read and write permissions by a single node.

   3

   The size of the persistent volume claim.

   4

   The name of the existing persistent volume.

5. Create the

   PersistentVolumeClaim

   object from the file:

   $ oc create -f pvc1.yaml

Procedure

1. Create the

   openshift-local-storage

   project:

   $ oc adm new-project openshift-local-storage

2. Optional: Allow local storage creation on infrastructure nodes.

   You might want to use the Local Storage Operator to create volumes on infrastructure nodes in support of components such as logging and monitoring.

   You must adjust the default node selector so that the Local Storage Operator includes the infrastructure nodes, and not just worker nodes.

   To block the Local Storage Operator from inheriting the cluster-wide default selector, enter the following command:

   $ oc annotate namespace openshift-local-storage openshift.io/node-selector=''

3. Optional: Allow local storage to run on the management pool of CPUs in single-node deployment.

   Use the Local Storage Operator in single-node deployments and allow the use of CPUs that belong to the

   management

   pool. Perform this step on single-node installations that use management workload partitioning.

   To allow Local Storage Operator to run on the management CPU pool, run following commands:

   $ oc annotate namespace openshift-local-storage workload.openshift.io/allowed='management'

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → OperatorHub.
3. Type Local Storage into the filter box to locate the Local Storage Operator.
4. Click Install.
5. On the Install Operator page, select A specific namespace on the cluster. Select openshift-local-storage from the drop-down menu.
6. Adjust the values for Update Channel and Approval Strategy to the values that you want.
7. Click Install.

From the CLI

1. Install the Local Storage Operator from the CLI.

   1. Create an object YAML file to define an Operator group and subscription for the Local Storage Operator, such as

      openshift-local-storage.yaml

      :

      Example openshift-local-storage.yaml

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: local-operator-group
        namespace: openshift-local-storage
      spec:
        targetNamespaces:
          - openshift-local-storage
      ---
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: local-storage-operator
        namespace: openshift-local-storage
      spec:
        channel: stable
        installPlanApproval: Automatic 

      1

      
        name: local-storage-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

      1

      The user approval policy for an install plan.

2. Create the Local Storage Operator object by entering the following command:

   $ oc apply -f openshift-local-storage.yaml

   At this point, the Operator Lifecycle Manager (OLM) is now aware of the Local Storage Operator. A ClusterServiceVersion (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

3. Verify local storage installation by checking that all pods and the Local Storage Operator have been created:

   1. Check that all the required pods have been created:

      $ oc -n openshift-local-storage get pods

      Example output

      NAME                                      READY   STATUS    RESTARTS   AGE
      local-storage-operator-746bf599c9-vlt5t   1/1     Running   0          19m

   2. Check the ClusterServiceVersion (CSV) YAML manifest to see that the Local Storage Operator is available in the

      openshift-local-storage

      project:

      $ oc get csvs -n openshift-local-storage

      Example output

      NAME                                         DISPLAY         VERSION               REPLACES   PHASE
      local-storage-operator.4.2.26-202003230335   Local Storage   4.2.26-202003230335              Succeeded

Procedure

1. Create the local volume resource. This resource must define the nodes and paths to the local volumes.

   Note

   Do not use different storage class names for the same device. Doing so will create multiple persistent volumes (PVs).

   Example: Filesystem

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 

   1

   
   spec:
     nodeSelector: 

   2

   
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-140-183
             - ip-10-0-158-139
             - ip-10-0-164-33
     storageClassDevices:
       - storageClassName: "local-sc" 

   3

   
         volumeMode: Filesystem 

   4

   
         fsType: xfs 

   5

   
         devicePaths: 

   6

   
           - /path/to/device 

   7

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   Note

   A raw block volume (

   volumeMode: Block

   ) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices.

   5

   The file system that is created when the local volume is mounted for the first time.

   6

   The path containing a list of local storage devices to choose from.

   7

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as /dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   Note

   If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the

   virsh edit <VM>

   command to add the

   <serial>mydisk</serial>

   definition.

   Example: Block

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 

   1

   
   spec:
     nodeSelector: 

   2

   
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-136-143
             - ip-10-0-140-255
             - ip-10-0-144-180
     storageClassDevices:
       - storageClassName: "local-sc" 

   3

   
         volumeMode: Block 

   4

   
         devicePaths: 

   5

   
           - /path/to/device 

   6

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   5

   The path containing a list of local storage devices to choose from.

   6

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   Note

   If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the

   virsh edit <VM>

   command to add the

   <serial>mydisk</serial>

   definition.

2. Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <local-volume>.yaml

3. Verify that the provisioner was created and that the corresponding daemon sets were created:

   $ oc get all -n openshift-local-storage

   Example output

   NAME                                          READY   STATUS    RESTARTS   AGE
   pod/diskmaker-manager-9wzms                   1/1     Running   0          5m43s
   pod/diskmaker-manager-jgvjp                   1/1     Running   0          5m43s
   pod/diskmaker-manager-tbdsj                   1/1     Running   0          5m43s
   pod/local-storage-operator-7db4bd9f79-t6k87   1/1     Running   0          14m
   
   NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
   service/local-storage-operator-metrics   ClusterIP   172.30.135.36   <none>        8383/TCP,8686/TCP   14m
   
   NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/diskmaker-manager   3         3         3       3            3           <none>          5m43s
   
   NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/local-storage-operator   1/1     1            1           14m
   
   NAME                                                DESIRED   CURRENT   READY   AGE
   replicaset.apps/local-storage-operator-7db4bd9f79   1         1         1       14m

   Note the desired and current number of daemon set processes. A desired count of

   0

   indicates that the label selectors were invalid.

4. Verify that the persistent volumes were created:

   $ oc get pv

   Example output

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-1cec77cf   100Gi      RWO            Delete           Available           local-sc                88m
   local-pv-2ef7cd2a   100Gi      RWO            Delete           Available           local-sc                82m
   local-pv-3fa1c73    100Gi      RWO            Delete           Available           local-sc                48m