Chapter 4. Dynamic provisioning

4.1. About dynamic provisioning

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any intimate knowledge about the underlying storage volume sources.

The OpenShift Container Platform persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OpenShift Container Platform. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plug-in APIs.

4.2. Available dynamic provisioning plug-ins

OpenShift Container Platform provides the following provisioner plug-ins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Storage type	Provisioner plug-in name	Notes
Red Hat OpenStack Platform (RHOSP) Cinder	`kubernetes.io/cinder`
AWS Elastic Block Store (EBS)	`kubernetes.io/aws-ebs`	For dynamic provisioning when using multiple clusters in different zones, tag each node with `Key=kubernetes.io/cluster/<cluster_name>,Value=<cluster_id>` where `<cluster_name>` and `<cluster_id>` are unique per cluster.
AWS Elastic File System (EFS)		Dynamic provisioning is accomplished through the EFS provisioner pod and not through a provisioner plug-in.
Azure Disk	`kubernetes.io/azure-disk`
Azure File	`kubernetes.io/azure-file`	The `persistent-volume-binder` ServiceAccount requires permissions to create and get Secrets to store the Azure storage account and keys.
GCE Persistent Disk (gcePD)	`kubernetes.io/gce-pd`	In multi-zone configurations, it is advisable to run one OpenShift Container Platform cluster per GCE project to avoid PVs from being created in zones where no node in the current cluster exists.
VMware vSphere	`kubernetes.io/vsphere-volume`

Important

Any chosen provisioner plug-in also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

4.3. Defining a StorageClass

StorageClass objects are currently a globally scoped object and must be created by cluster-admin or storage-admin users.

Important

The ClusterStorageOperator may install a default StorageClass depending on the platform in use. This StorageClass is owned and controlled by the operator. It cannot be deleted or modified beyond defining annotations and labels. If different behavior is desired, you must define a custom StorageClass.

The following sections describe the basic object definition for a StorageClass and specific examples for each of the supported plug-in types.

4.3.1. Basic StorageClass object definition

The following resource shows the parameters and default values that you use to configure a StorageClass. This example uses the AWS ElasticBlockStore (EBS) object definition.

Sample StorageClass definition

kind: StorageClass 1
apiVersion: storage.k8s.io/v1 2
metadata:
  name: gp2 3
  annotations: 4
    storageclass.kubernetes.io/is-default-class: 'true'
    ...
provisioner: kubernetes.io/aws-ebs 5
parameters: 6
  type: gp2
...

1: (required) The API object type.
2: (required) The current apiVersion.
3: (required) The name of the StorageClass.
4: (optional) Annotations for the StorageClass
5: (required) The type of provisioner associated with this storage class.
6: (optional) The parameters required for the specific provisioner, this will change from plug-in to plug-in.

4.3.2. StorageClass annotations

To set a StorageClass as the cluster-wide default, add the following annotation to your StorageClass’s metadata:

storageclass.kubernetes.io/is-default-class: "true"

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
...

This enables any Persistent Volume Claim (PVC) that does not specify a specific volume to automatically be provisioned through the default StorageClass.

Note

The beta annotation storageclass.beta.kubernetes.io/is-default-class is still working; however, it will be removed in a future release.

To set a StorageClass description, add the following annotation to your StorageClass’s metadata:

kubernetes.io/description: My StorageClass Description

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubernetes.io/description: My StorageClass Description
...

4.3.3. RHOSP Cinder object definition

cinder-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gold
provisioner: kubernetes.io/cinder
parameters:
  type: fast  1
  availability: nova 2
  fsType: ext4 3

1: Volume type created in Cinder. Default is empty.
2: Availability Zone. If not specified, volumes are generally round-robined across all active zones where the OpenShift Container Platform cluster has a node.
3: File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

4.3.4. AWS Elastic Block Store (EBS) object definition

aws-ebs-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 1
  iopsPerGB: "10" 2
  encrypted: "true" 3
  kmsKeyId: keyvalue 4
  fsType: ext4 5

1: (required) Select from io1, gp2, sc1, st1. The default is gp2. See the AWS documentation for valid Amazon Resource Name (ARN) values.
2: (optional) Only for io1 volumes. I/O operations per second per GiB. The AWS volume plug-in multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See the AWS documentation for further details.
3: (optional) Denotes whether to encrypt the EBS volume. Valid values are true or false.
4: (optional) The full ARN of the key to use when encrypting the volume. If none is supplied, but encypted is set to true, then AWS generates a key. See the AWS documentation for a valid ARN value.
5: (optional) File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

4.3.5. Azure Disk object definition

azure-advanced-disk-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: azure_storage_account_name  1
  storageaccounttype: Standard_LRS  2
  kind: Dedicated  3

1

Azure storage account name. This must reside in the same resource group as the cluster. If a storage account is specified, the location is ignored. If a storage account is not specified, a new storage account gets created in the same resource group as the cluster. If you are specifying a storageAccount, the value for kind must be Dedicated.

2

Azure storage account SKU tier. Default is empty. Note that Premium VMs can attach both Standard_LRS and Premium_LRS disks, Standard VMs can only attach Standard_LRS disks, Managed VMs can only attach managed disks, and unmanaged VMs can only attach unmanaged disks.

3

Possible values are Shared (default), Dedicated, and Managed.

If kind is set to Shared, Azure creates all unmanaged disks in a few shared storage accounts in the same resource group as the cluster.
If kind is set to Managed, Azure creates new managed disks.
If kind is set to Dedicated and a storageAccount is specified, Azure uses the specified storage account for the new unmanaged disk in the same resource group as the cluster. For this to work:
- The specified storage account must be in the same region.
- Azure Cloud Provider must have a write access to the storage account.
If kind is set to Dedicated and a storageAccount is not specified, Azure creates a new dedicated storage account for the new unmanaged disk in the same resource group as the cluster.

4.3.6. Azure File object definition

The Azure File StorageClass uses secrets to store the Azure storage account name and the storage account key that are required to create an Azure Files share. These permissions are created as part of the following procedure.

Procedure

Define a ClusterRole that allows access to create and view secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
#  name: system:azure-cloud-provider
  name: <persistent-volume-binder-role> 1
rules:
- apiGroups: ['']
  resources: ['secrets']
  verbs:     ['get','create']

1: The name of the ClusterRole to view and create secrets.

Add the ClusterRole to the ServiceAccount:

$ oc adm policy add-cluster-role-to-user <persistent-volume-binder-role> system:serviceaccount:kube-system:persistent-volume-binder

Create the Azure File StorageClass:
```
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <azure-file> 1
provisioner: kubernetes.io/azure-file
parameters:
  location: eastus 2
  skuName: Standard_LRS 3
  storageAccount: <storage-account> 4
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
1
Name of the StorageClass. The PersistentVolumeClaim uses this StorageClass for provisioning the associated PersistentVolumes.
2
Location of the Azure storage account, such as eastus. Default is empty, meaning that a new Azure storage account will be created in the OpenShift Container Platform cluster’s location.
3
SKU tier of the Azure storage account, such as Standard_LRS. Default is empty, meaning that a new Azure storage account will be created with the Standard_LRS SKU.
4
Name of the Azure storage account. If a storage account is provided, then skuName and location are ignored. If no storage account is provided, then the StorageClass searches for any storage account that is associated with the resource group for any accounts that match the defined skuName and location.

4.3.6.1. Considerations when using Azure File

The following file system features are not supported by the default Azure File StorageClass:

Symlinks
Hard links
Extended attributes
Sparse files
Named pipes

Additionally, the owner user identifier (UID) of the Azure File mounted directory is different from the process UID of the container. The uid mount option can be specified in the StorageClass to define a specific user identifier to use for the mounted directory.

The following StorageClass demonstrates modifying the user and group identifier, along with enabling symlinks for the mounted directory.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-file
mountOptions:
  - uid=1500 1
  - gid=1500 2
  - mfsymlinks 3
provisioner: kubernetes.io/azure-file
parameters:
  location: eastus
  skuName: Standard_LRS
reclaimPolicy: Delete
volumeBindingMode: Immediate

1: Specifies the user identifier to use for the mounted directory.
2: Specifies the group identifier to use for the mounted directory.
3: Enables symlinks.

4.3.7. GCE PersistentDisk (gcePD) object definition

gce-pd-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard  1
  replication-type: none

1: Select either pd-standard or pd-ssd. The default is pd-ssd.

4.3.8. VMware vSphere object definition

vsphere-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/vsphere-volume 1
parameters:
  diskformat: thin 2

1: For more information about using VMware vSphere with OpenShift Container Platform, see the VMware vSphere documentation.
2: diskformat: thin, zeroedthick and eagerzeroedthick are all valid disk formats. See vSphere docs for additional details regarding the disk format types. The default value is thin.

4.3.9. Red Hat OpenShift Container Storage object definition

When using Red Hat OpenShift Container Storage, the storage classes for dynamic volume provisioning are created when Red Hat OpenShift Container Storage 4.2 is deployed from the Operator Hub as described in Verify that the storage classes are created and listed.

4.4. Changing the default StorageClass

If you are using AWS, use the following process to change the default StorageClass. This process assumes you have two StorageClasses defined, gp2 and standard, and you want to change the default StorageClass from gp2 to standard.

List the StorageClass:

$ oc get storageclass

NAME                 TYPE
gp2 (default)        kubernetes.io/aws-ebs 1
standard             kubernetes.io/aws-ebs

1: (default) denotes the default StorageClass.

Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default StorageClass:

$ oc patch storageclass gp2 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

Make another StorageClass the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.
```
$ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'
```

Verify the changes:

$ oc get storageclass

NAME                 TYPE
gp2                  kubernetes.io/aws-ebs
standard (default)   kubernetes.io/aws-ebs