Chapter 1. Deploying OpenShift Container Storage

Procedure

1. Click Operators OperatorHub in the left pane of the OpenShift Web Console.

   Figure 1.1. List of operators in the Operator Hub

   
2. Search for OpenShift Container Storage Operator from the list of operators and click on it.
3. On the OpenShift Container Storage Operator page, click Install.

4. On the Create Operator Subscription page, the Installation Mode, Update Channel, and Approval Strategy options can be set.

   Figure 1.2. Create Operator Subscription page

   

   1. Select A specific namespace on the cluster for the Installation Mode option.

      • Select openshift-storage namespace from the drop down menu.
   2. Select stable-4.3 as the update channel based on your requirement.

   3. Select an Approval Strategy:

      • Automatic specifies that you want OpenShift Container Platform to upgrade OpenShift Container Storage automatically.
      • Manual specifies that you want to have control to upgrade OpenShift Container Storage manually.

5. Click Subscribe.

   Figure 1.3. Installed operators

   

   The Installed Operators page is displayed with the status of the operator.

Procedure

1. Click Operators Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. On the Installed Operator page, select openshift-storage from the Project drop down list to switch to the openshift-storage project.

3. Click OpenShift Container Storage Operator.

   OpenShift Container Storage operator creates a OCSInitialization resource automatically.

4. On the OpenShift Container Storage Operator page, scroll right and click the Storage Cluster tab.

   Figure 1.4. OpenShift Container Storage Operator page

   

5. On the OCS Cluster Services page, click Create OCS Cluster Service.

   Figure 1.5. Create New OCS Service page

   

6. On the Create New OCS Service page, perform the following:

   1. Select at least three worker nodes from the available list of nodes for the use of OpenShift Container Storage service. Ensure that the nodes are in different Location.
   2. Storage Class is set by default to gp2 for AWS and thin for VMware.

   3. Select OCS Service Capacity from drop down list.

      Note

      Once you select the initial storage capacity here, you can add more capacity only in this increment.

7. Click Create.

   The Create button is enabled only after you select three nodes. A new storage cluster of three volumes will be created with one volume per worker node. The default configuration uses a replication factor of 3.

1. Log in to the Red Hat Enterprise Linux based node and open a terminal.

2. Verify that the node has access to the rhel-7-server-extras-rpms repository.

   # subscription-manager repos --list-enabled | grep rhel-7-server

   If you do not see both rhel-7-server-rpms and rhel-7-server-extras-rpms in the output, or if there is no output, run the following commands to enable each repository.

   # subscription-manager repos --enable=rhel-7-server-rpms
   # subscription-manager repos --enable=rhel-7-server-extras-rpms

3. Install the required packages.

   # yum install -y policycoreutils container-selinux

4. Persistently enable container use of the Ceph file system in SELinux.

   # setsebool -P container_use_cephfs on

5. Verify that containers can now access OpenShift Container Storage hosted on this node.

Procedure

1. Click Operators OperatorHub in the left pane of the OpenShift Web Console.

   Figure 1.6. List of operators in the Operator Hub

   
2. Search for OpenShift Container Storage Operator from the list of operators and click on it.
3. On the OpenShift Container Storage Operator page, click Install.

4. On the Create Operator Subscription page, the Installation Mode, Update Channel, and Approval Strategy options can be set.

   Figure 1.7. Create Operator Subscription page

   

   1. Select A specific namespace on the cluster for the Installation Mode option.

      • Select openshift-storage namespace from the drop down menu.
   2. Select stable-4.3 as the update channel based on your requirement.

   3. Select an Approval Strategy:

      • Automatic specifies that you want OpenShift Container Platform to upgrade OpenShift Container Storage automatically.
      • Manual specifies that you want to have control to upgrade OpenShift Container Storage manually.

5. Click Subscribe.

   Figure 1.8. Installed operators

   

   The Installed Operators page is displayed with the status of the operator.

Procedure

1. Click Operators OperatorHub in the left pane of the OpenShift Web Console.
2. Search for Local Storage Operator from the list of operators and click on it.

3. Click Install.

   Figure 1.9. Create Operator Subscription page

   

4. Select A specific namespace on the cluster for the Installation Mode option.

   • Select local-storage namespace from the drop down menu.
5. Select a desired value for the Update Channel option.
6. Select the desired Approval Strategy.
7. Click Subscribe.
8. Verify that the Local Storage Operator show the Status as Succeeded.

Procedure

1. List and verify the name of the worker nodes with the OpenShift Container Storage label.

   $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage=

   Example output:

   NAME                                        STATUS   ROLES    AGE     VERSION
   ip-10-0-135-71.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2
   ip-10-0-145-125.us-east-2.compute.internal   Ready    worker   6h45m   v1.16.2
   ip-10-0-160-91.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2

2. Log in to each worker node that is used for OpenShift Container Storage resources and find the unique by-id device name for each available raw block device.

   $ oc debug node/<Nodename>

   Example output:

   $ oc debug node/ip-10-0-135-71.us-east-2.compute.internal
   Starting pod/ip-10-0-135-71us-east-2computeinternal-debug ...
   To use host binaries, run `chroot /host`
   Pod IP: 10.0.135.71
   If you don't see a command prompt, try pressing enter.
   sh-4.2# chroot /host
   sh-4.4# lsblk
   NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   xvda                         202:0    0   120G  0 disk
   |-xvda1                      202:1    0   384M  0 part /boot
   |-xvda2                      202:2    0   127M  0 part /boot/efi
   |-xvda3                      202:3    0     1M  0 part
   `-xvda4                      202:4    0 119.5G  0 part
     `-coreos-luks-root-nocrypt 253:0    0 119.5G  0 dm   /sysroot
   nvme0n1                      259:0    0   1.7T  0 disk
   nvme1n1                      259:1    0   1.7T  0 disk

   In this example, the local devices available are nvme0n1 and nvme1n1.

3. Find the unique by-id device name depending on the hardware serial number for each device.

   sh-4.4# ls -l /dev/disk/by-id/
   total 0
   lrwxrwxrwx. 1 root root 10 Mar 17 16:24 dm-name-coreos-luks-root-nocrypt -> ../../dm-0
   lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC -> ../../nvme0n1
   lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS60382E5D7441494EC -> ../../nvme1n1
   lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-nvme.1d0f-4157533130333832453544373434313439344543-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme0n1
   lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-nvme.1d0f-4157533630333832453544373434313439344543-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme1n1

   In this example, all the OpenShift Container Storage worker nodes are of the Amazon EC2 type i3.2xlarge. So, all the three worker nodes have the same type of machine but the by-id identifier is unique for every local device. The lsblk command shows the last two devices, nvme0n1 and nvme1n1 with a size of 1.7 TB each.

   For each worker node that has the OpenShift Container Storage label (a minimum of three), you need to find the unique by-id device names. In this example, the by-id device names are:

   • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
   • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS60382E5D7441494EC

1. Creating PVs by using the LocalVolume CR
2. Creating a new StorageClass

Procedure

1. Create local persistent volumes (PVs) on the storage nodes using LocalVolume custom resource (CR).

   Example of LocalVolume CR local-storage-block.yaml using OpenShift Storage Container label as node selector and by-id device identifier:

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     nodeSelector:
       nodeSelectorTerms:
         - matchExpressions:
             - key: cluster.ocs.openshift.io/openshift-storage
               operator: In
               values:
                 - ''
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8   # <-- modify this line

   Each Amazon EC2 instance has two disks and this example uses both the disks.

2. Create the LocalVolume CR.

   $ oc create -f local-storage-block.yaml

   Example output:

   localvolume.local.storage.openshift.io/local-block created

3. Check if the pods are created.

   $ oc -n local-storage get pods

   Example output:

   NAME                                      READY   STATUS    RESTARTS   AGE
   local-block-local-diskmaker-59rmn         1/1     Running   0          15m
   local-block-local-diskmaker-6n7ct         1/1     Running   0          15m
   local-block-local-diskmaker-jwtsn         1/1     Running   0          15m
   local-block-local-provisioner-6ssxc       1/1     Running   0          15m
   local-block-local-provisioner-swwvx       1/1     Running   0          15m
   local-block-local-provisioner-zmv5j       1/1     Running   0          15m
   local-storage-operator-7848bbd595-686dg   1/1     Running   0          15m

4. Check if the PVs are created.

   You must see a new PV for each of the local storage devices on the three worker nodes. Refer the example in the Finding available storage devices section that shows two available storage devices per worker node with a size 1.7 TB for each node.

   $ oc get pv

   Example output:

   NAME               CAPACITY ACCESS MODES  RECLAIM POLICY STATUS     CLAIM   STORAGECLASS  REASON   AGE
   local-pv-1a46bc79  2328Gi   RWO           Delete         Available          localblock             14m
   local-pv-429d90ee  2328Gi   RWO           Delete         Available          localblock             14m
   local-pv-4d0a62e3  2328Gi   RWO           Delete         Available          localblock             14m
   local-pv-55c05d76  2328Gi   RWO           Delete         Available          localblock             14m
   local-pv-5c7b0990  2328Gi   RWO           Delete         Available          localblock             14m
   local-pv-a6b283b   2328Gi   RWO           Delete         Available          localblock             14m

5. Check if a new StorageClass is created due to the creation of LocalVolume CR. This StorageClass is used while creating StorageCluster to create PVCs.

   $ oc get sc | grep localblock

   Example output:

   NAME            PROVISIONER                    AGE
   localblock      kubernetes.io/no-provisioner   7m46s

6. Create StorageCluster CR that uses the localblock StorageClass and the three PVs that are created.

   Example StorageCluster CR ocs-cluster-service.yaml using monDataDirHostPath and localblock StorageClass.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     resources:
       mds:
         limits:
           cpu: 3
         requests:
           cpu: 1
       noobaa-core:
         limits:
           cpu: 2
           memory: 8Gi
         requests:
           cpu: 1
           memory: 8Gi
       noobaa-db:
         limits:
           cpu: 2
           memory: 8Gi
         requests:
           cpu: 1
           memory: 8Gi
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
       - count: 2
         dataPVCTemplate:
           spec:
             accessModes:
               - ReadWriteOnce
             resources:
               requests:
                 storage: 2328Gi
             storageClassName: localblock
             volumeMode: Block
         name: ocs-deviceset
         placement: {}
         portable: false
         replica: 3
         resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

7. Create StorageCluster CR.

   $ oc create -f ocs-cluster-service.yaml

   Example output

   storagecluster.ocs.openshift.io/ocs-cluster-service created

Procedure

1. Create the LocalVolume CR for block PVs.

   Example of LocalVolume CR local-storage-block.yaml using OpenShift Container Storage label as node:

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     nodeSelector:
       nodeSelectorTerms:
       - matchExpressions:
           - key: cluster.ocs.openshift.io/openshift-storage
             operator: In
             values:
             - ""
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/scsi-36000c2991c27c2e5ba7c47d1e4352de2   # <-- modify this line
           - /dev/disk/by-id/scsi-36000c29682ca9e347926406711f3dc4e   # <-- modify this line
           - /dev/disk/by-id/scsi-36000c296aaf03a9b1e4b01d086bc6348   # <-- modify this line

2. Create LocalVolume CR for block PVs.

   $ oc create -f local-storage-block.yaml

   Example output:

   localvolume.local.storage.openshift.io/local-block created

3. Check if the pods are created.

   $ oc -n local-storage get pods

   Example output:

   NAME                                      READY   STATUS    RESTARTS   AGE
   local-block-local-diskmaker-5brzv         1/1     Running   0          31s
   local-block-local-diskmaker-8sxcs         1/1     Running   0          31s
   local-block-local-diskmaker-s7s9p         1/1     Running   0          31s
   local-block-local-provisioner-9cbw8       1/1     Running   0          31s
   local-block-local-provisioner-cpddv       1/1     Running   0          31s
   local-block-local-provisioner-f6h7h       1/1     Running   0          31s
   local-storage-operator-75b9776b75-vwdzh   1/1     Running   0          2m47s

4. Check the new localblock StorageClass.

   $ oc get sc | grep localblock

   Example output:

   NAME            PROVISIONER                     AGE
   localblock      kubernetes.io/no-provisioner    8m38s

5. Check the PVs that are created with the Available status.

   $ oc get pv

   Example output:

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-150fdc87   100Gi      RWO            Delete           Available           localblock              2m11s
   local-pv-183bfc0a   100Gi      RWO            Delete           Available           localblock              2m11s
   local-pv-b2f5cb25   100Gi      RWO            Delete           Available           localblock              2m21s

   In this example, three PVs are used for OSD storage (100 GB).

6. Create StorageCluster CR ocs-cluster-service-VMware.yaml that uses the monDataDirHostPath and localblock StorageClass.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
     - count: 1
       dataPVCTemplate:
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 100Gi
           storageClassName: localblock
           volumeMode: Block
       name: ocs-deviceset
       placement: {}
       portable: false
       replica: 3
       resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

7. Create StorageCluster CR.

   $ oc create -f ocs-cluster-service-VMware.yaml

   Example output:

   storagecluster.ocs.openshift.io/ocs-storagecluster created

Procedure

1. Create LocalVolume CR for block PVs.

   Example of LocalVolume CR local-storage-block.yaml using OCS label as node selector.

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     nodeSelector:
       nodeSelectorTerms:
       - matchExpressions:
           - key: cluster.ocs.openshift.io/openshift-storage
             operator: In
             values:
             - ""
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY81260978128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY80440W5U128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB85AABDE128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0A60CB81128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0093D45E128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYE46F6060128A   # <-- modify this line

2. Create the LocalVolume CR for block PVs.

   $ oc create -f local-storage-block.yaml

3. Check if the pods are created.

   $ oc -n local-storage get pods

4. Check if the PVs are created.

   $ oc get pv

5. Check for the new localblock StorageClass.

   $ oc get sc | grep localblock

   Example output:

   NAME            PROVISIONER                     AGE
   localblock     kubernetes.io/no-provisioner    10m20s

6. Create StorageCluster CR cluster-service-metal.yaml using monDataDirHostPath and localblock storage classes.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
     - count: 2
       dataPVCTemplate:
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 2Ti
           storageClassName: localblock
           volumeMode: Block
       name: ocs-deviceset
       placement: {}
       portable: false
       replica: 3
       resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

7. Create the StorageCluster CR.

   $ oc create -f cluster-service-metal.yaml

   Example output:

   storagecluster.ocs.openshift.io/ocs-storagecluster created