Managing and allocating storage resources

Procedure

1. Click Storage → StorageClasses.
2. Click Create Storage Class.
3. Enter the storage class Name and Description.

4. Reclaim Policy is set to Delete as the default option. Use this setting.

   If you change the reclaim policy to Retain in the storage class, the persistent volume (PV) remains in Released state even after deleting the persistent volume claim (PVC).

5. Volume binding mode is set to WaitForConsumer as the default option.

   If you choose the Immediate option, then the PV gets created immediately when creating the PVC.

6. Select RBD or CephFS Provisioner as the plugin for provisioning the persistent volumes.

7. Select an existing Storage Pool from the list or create a new pool.

   Note

   The 2-way replication data protection policy is only supported for the non-default RBD pool. 2-way replication can be used by creating an additional pool. To know about Data Availability and Integrity considerations for replica 2 pools, see Knowledgebase Customer Solution Article.

   Create new pool

   1. Click Create New Pool.
   2. Enter Pool name.
   3. Choose 2-way-Replication or 3-way-Replication as the Data Protection Policy.

   4. Select Enable compression if you need to compress the data.

      Enabling compression can impact application performance and might prove ineffective when data to be written is already compressed or encrypted. Data written before enabling compression will not be compressed.

   5. Click Create to create the new storage pool.
   6. Click Finish after the pool is created.

8. Optional: Select Enable Encryption checkbox.
9. Click Create to create the storage class.

Procedure

1. Click Storage → Data Foundation.
2. In the Storage systems tab, select the storage system and then click the BlockPools tab.
3. Click Create Block Pool.

4. Enter Pool name.

   Note

   Using 2-way replication data protection policy is not supported for the default pool. However, you can use 2-way replication if you are creating an additional pool.

5. Select Data protection policy as either 2-way Replication or 3-way Replication.
6. Select Volume Type.

7. Optional: Select Enable compression checkbox if you need to compress the data.

   Enabling compression can impact application performance and might prove ineffective when data to be written is already compressed or encrypted. Data written before enabling compression will not be compressed.

8. Click Create.

Procedure

1. Click Storage → Data Foundation.
2. In the Storage systems tab, select the storage system and then click BlockPools.
3. Click the Action Menu (⋮) at the end the pool you want to update.
4. Click Edit Block Pool.

5. Modify the form details as follows:

   Note

   Using 2-way replication data protection policy is not supported for the default pool. However, you can use 2-way replication if you are creating an additional pool.

   1. Change the Data protection policy to either 2-way Replication or 3-way Replication.

   2. Enable or disable the compression option.

      Enabling compression can impact application performance and might prove ineffective when data to be written is already compressed or encrypted. Data written before enabling compression will not be compressed.

6. Click Save.

Procedure

1. . Click Storage → Data Foundation.
2. In the Storage systems tab, select the storage system and then click the BlockPools tab.
3. Click the Action Menu (⋮) at the end the pool you want to delete.
4. Click Delete Block Pool.
5. Click Delete to confirm the removal of the Pool.

Procedure

1. Create a Persistent Volume Claim for the Image Registry to use.

   1. In the OpenShift Web Console, click Storage → Persistent Volume Claims.
   2. Set the Project to openshift-image-registry.

   3. Click Create Persistent Volume Claim.

      1. From the list of available storage classes retrieved above, specify the Storage Class with the provisioner openshift-storage.cephfs.csi.ceph.com.
      2. Specify the Persistent Volume Claim Name, for example, ocs4registry.
      3. Specify an Access Mode of Shared Access (RWX).
      4. Specify a Size of at least 100 GB.

      5. Click Create.

         Wait until the status of the new Persistent Volume Claim is listed as Bound.

2. Configure the cluster’s Image Registry to use the new Persistent Volume Claim.

   1. Click Administration → Custom Resource Definitions.
   2. Click the Config custom resource definition associated with the imageregistry.operator.openshift.io group.
   3. Click the Instances tab.
   4. Beside the cluster instance, click the Action Menu (⋮) → Edit Config.

   5. Add the new Persistent Volume Claim as persistent storage for the Image Registry.

      1. Add the following under spec:, replacing the existing storage: section if necessary.

           storage:
             pvc:
               claim: <new-pvc-name>

         For example:

           storage:
             pvc:
               claim: ocs4registry

      2. Click Save.

3. Verify that the new configuration is being used.

   1. Click Workloads → Pods.
   2. Set the Project to openshift-image-registry.
   3. Verify that the new image-registry-* pod appears with a status of Running, and that the previous image-registry-* pod terminates.
   4. Click the new image-registry-* pod to view pod details.
   5. Scroll down to Volumes and verify that the registry-storage volume has a Type that matches your new Persistent Volume Claim, for example, ocs4registry.

Procedure

1. Create ObjectBucketClaim by following the steps in Creating Object Bucket Claim.

2. Create an image-registry-private-configuration-user secret.

   1. Go to the OpenShift web-console.
   2. Click ObjectBucketClaim -→ ObjectBucketClaim Data.
   3. In the ObjectBucketClaim data, look for MCG access key and MCG secret key in the openshift-image-registry namespace.

   4. Create the secret using the following command:

      $ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=<MCG Accesskey> --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=<MCG Secretkey> --namespace openshift-image-registry

3. Change the status of managementState of Image Registry Operator to Managed.

   $ oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec": {"managementState": "Managed"}}'

4. Edit the spec.storage section of Image Registry Operator configuration file:

   1. Get the unique-bucket-name and regionEndpoint under the Object Bucket Claim Data section from the Web Console OR you can also get the information on regionEndpoint and unique-bucket-name from the command:

       $ oc describe  noobaa

   2. Add regionEndpoint as http://<Endpoint-name>:<port> if the

      • storageclass is ceph-rgw storageclass and the
      • endpoint points to the internal SVC from the openshift-storage namespace.

   3. An image-registry pod spawns after you make the changes to the Operator registry configuration file.

      $ oc edit configs.imageregistry.operator.openshift.io -n openshift-image-registry
      apiVersion: imageregistry.operator.openshift.io/v1
      kind: Config
      metadata:
      [..]
      name: cluster
      spec:
      [..]
      storage:
      s3:
      
        bucket: <Unique-bucket-name>
      
        region: us-east-1 (Use this region as default)
      
        regionEndpoint: https://<Endpoint-name>:<port>
      
        virtualHostedStyle: false

5. Reset the image registry settings to default.

   $ oc get pods -n openshift-image-registry

Procedure

1. In the OpenShift Web Console, go to Workloads → Config Maps.
2. Set the Project dropdown to openshift-monitoring.
3. Click Create Config Map.

4. Define a new cluster-monitoring-config Config Map using the following example.

   Replace the content in angle brackets (<, >) with your own values, for example, retention: 24h or storage: 40Gi.

   Replace the storageClassName with the storageclass that uses the provisioner openshift-storage.rbd.csi.ceph.com. In the example given below the name of the storageclass is ocs-storagecluster-ceph-rbd.

   Example cluster-monitoring-config Config Map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
         prometheusK8s:
           retention: <time to retain monitoring files, e.g. 24h>
           volumeClaimTemplate:
             metadata:
               name: ocs-prometheus-claim
             spec:
               storageClassName: ocs-storagecluster-ceph-rbd
               resources:
                 requests:
                   storage: <size of claim, e.g. 40Gi>
         alertmanagerMain:
           volumeClaimTemplate:
             metadata:
               name: ocs-alertmanager-claim
             spec:
               storageClassName: ocs-storagecluster-ceph-rbd
               resources:
                 requests:
                   storage: <size of claim, e.g. 40Gi>

5. Click Create to save and create the Config Map.

Verification steps

1. Verify that the Persistent Volume Claims are bound to the pods.

   1. Go to Storage → Persistent Volume Claims.
   2. Set the Project dropdown to openshift-monitoring.

   3. Verify that 5 Persistent Volume Claims are visible with a state of Bound, attached to three alertmanager-main-* pods, and two prometheus-k8s-* pods.

      Figure 4.1. Monitoring storage created and bound

      

2. Verify that the new alertmanager-main-* pods appear with a state of Running.

   1. Go to Workloads → Pods.
   2. Click the new alertmanager-main-* pods to view the pod details.

   3. Scroll down to Volumes and verify that the volume has a Type, ocs-alertmanager-claim that matches one of your new Persistent Volume Claims, for example, ocs-alertmanager-claim-alertmanager-main-0.

      Figure 4.2. Persistent Volume Claims attached to alertmanager-main-* pod

      

3. Verify that the new prometheus-k8s-* pods appear with a state of Running.

   1. Click the new prometheus-k8s-* pods to view the pod details.

   2. Scroll down to Volumes and verify that the volume has a Type, ocs-prometheus-claim that matches one of your new Persistent Volume Claims, for example, ocs-prometheus-claim-prometheus-k8s-0.

      Figure 4.3. Persistent Volume Claims attached to prometheus-k8s-* pod

      

Procedure

1. Deploy storagecluster either from the command line interface or the user interface.

2. Label the application namespace.

   apiVersion: v1
   kind: Namespace
   metadata:
        name: <desired_name>
        labels:
             storagequota: <desired_label>

   <desired_name>

   Specify a name for the application namespace, for example, quota-rbd.

   <desired_label>

   Specify a label for the storage quota, for example, storagequota1.

3. Edit the storagecluster to set the quota limit on the storage class.

   $ oc edit storagecluster -n openshift-storage <ocs_storagecluster_name>

   <ocs_storagecluster_name>

   Specify the name of the storage cluster.

4. Add an entry for Overprovision Control with the desired hard limit into the StorageCluster.Spec:

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   spec:
    [...]
        overprovisionControl:
        - capacity: <desired_quota_limit>
             storageClassName: <storage_class_name>
             quotaName: <desired_quota_name>
             selector:
               labels:
                      matchLabels:
                       storagequota: <desired_label>
   [...]

   <desired_quota_limit>

   Specify a desired quota limit for the storage class, for example, 27Ti.

   <storage_class_name>

   Specify the name of the storage class for which you want to set the quota limit, for example, ocs-storagecluster-ceph-rbd.

   <desired_quota_name>

   Specify a name for the storage quota, for example, quota1.

   <desired_label>

   Specify a label for the storage quota, for example, storagequota1.

5. Save the modified storagecluster.

6. Verify that the clusterresourcequota is defined.

   Note

   Expect the clusterresourcequota with the quotaName that you defined in the previous step, for example, quota1.

   $ oc get clusterresourcequota -A
   
   $ oc describe clusterresourcequota -A

Procedure

1. In the OpenShift Web Console, click Compute → Nodes, and then select the node which has to be tainted.
2. In the Details page click on Edit taints.
3. Enter the values in the Key <node.ocs.openshift.io/storage>, Value <true> and in the Effect<Noschedule> field.
4. Click Save.

Procedure

1. Create a Persistent Volume Claim (PVC) for the application to use.

   1. In OpenShift Web Console, click Storage → Persistent Volume Claims.
   2. Set the Project for the application pod.

   3. Click Create Persistent Volume Claim.

      1. Specify a Storage Class provided by OpenShift Data Foundation.
      2. Specify the PVC Name, for example, myclaim.

      3. Select the required Access Mode.

         Note

         The Access Mode, Shared access (RWX) is not supported in IBM FlashSystem.

      4. For Rados Block Device (RBD), if the Access mode is ReadWriteOnce (RWO), select the required Volume mode. The default volume mode is Filesystem.
      5. Specify a Size as per application requirement.
      6. Click Create and wait until the PVC is in Bound status.

2. Configure a new or existing application pod to use the new PVC.

   • For a new application pod, perform the following steps:

     1. Click Workloads →Pods.
     2. Create a new application pod.

     3. Under the spec: section, add volumes: section to add the new PVC as a volume for the application pod.

        volumes:
          - name: <volume_name>
            persistentVolumeClaim:
              claimName: <pvc_name>

        For example:

        volumes:
          - name: mypd
            persistentVolumeClaim:
              claimName: myclaim

   • For an existing application pod, perform the following steps:

     1. Click Workloads →Deployment Configs.
     2. Search for the required deployment config associated with the application pod.
     3. Click on its Action menu (⋮) → Edit Deployment Config.

     4. Under the spec: section, add volumes: section to add the new PVC as a volume for the application pod and click Save.

        volumes:
          - name: <volume_name>
            persistentVolumeClaim:
              claimName: <pvc_name>

        For example:

        volumes:
          - name: mypd
            persistentVolumeClaim:
              claimName: myclaim

3. Verify that the new configuration is being used.

   1. Click Workloads → Pods.
   2. Set the Project for the application pod.
   3. Verify that the application pod appears with a status of Running.
   4. Click the application pod name to view pod details.
   5. Scroll down to Volumes section and verify that the volume has a Type that matches your new Persistent Volume Claim, for example, myclaim.

Procedure

1. Log in to OpenShift Web Console.
2. Click Storage → Persistent Volume Claims
3. Search for the required PVC name by using the Filter textbox. You can also filter the list of PVCs by Name or Label to narrow down the list
4. Check the Status column corresponding to the required PVC.
5. Click the required Name to view the PVC details.

Procedure

1. In the OpenShift Web Console, click Storage → Data Foundation.
2. In the Storage systems tab, select the storage system and then click Overview → Block and File.
3. Locate the Inventory card to see the number of PVCs with errors.
4. Click Storage → Persistent Volume Claims
5. Search for the required PVC using the Filter textbox.
6. Click on the PVC name and navigate to Events
7. Address the events as required or as directed.

Procedure

1. In OpenShift Web Console, navigate to Storage → Persistent Volume Claims.
2. Click the Action Menu (⋮) next to the Persistent Volume Claim you want to expand.

3. Click Expand PVC:

   

4. Select the new size of the Persistent Volume Claim, then click Expand:

   

5. To verify the expansion, navigate to the PVC’s details page and verify the Capacity field has the correct size requested.

   Note

   When expanding PVCs based on Ceph RADOS Block Devices (RBDs), if the PVC is not already attached to a pod the Condition type is FileSystemResizePending in the PVC’s details page. Once the volume is mounted, filesystem resize succeeds and the new size is reflected in the Capacity field.

Procedure

1. Get the persistent volume claim (PVC) details.

   $ oc get pvc data-pvc

   NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                          AGE
   data-pvc  Bound    pvc-f37b8582-4b04-4676-88dd-e1b95c6abf74   1Gi        RWO            ocs-storagecluster-ceph-rbd           20h

2. Add annotation reclaimspace.csiaddons.openshift.io/schedule=@monthly to the PVC to create reclaimspacecronjob.

   $ oc annotate pvc data-pvc "reclaimspace.csiaddons.openshift.io/schedule=@monthly"

   persistentvolumeclaim/data-pvc annotated

3. Verify that reclaimspacecronjob is created in the format, "<pvc-name>-xxxxxxx".

   $ oc get reclaimspacecronjobs.csiaddons.openshift.io

   NAME                    SCHEDULE    SUSPEND   ACTIVE   LASTSCHEDULE   AGE
   data-pvc-1642663516     @monthly                                      3s

4. Modify the schedule to run this job automatically.

   $ oc annotate pvc data-pvc "reclaimspace.csiaddons.openshift.io/schedule=@weekly" --overwrite=true

   persistentvolumeclaim/data-pvc annotated

5. Verify that the schedule for reclaimspacecronjob has been modified.

   $ oc get reclaimspacecronjobs.csiaddons.openshift.io

   NAME                  SCHEDULE    SUSPEND   ACTIVE   LASTSCHEDULE   AGE
   data-pvc-1642664617   @weekly                                       3s

Procedure

1. Create and apply the following custom resource for reclaim space operation:

   apiVersion: csiaddons.openshift.io/v1alpha1
   kind: ReclaimSpaceJob
   metadata:
     name: sample-1
   spec:
     target:
       persistentVolumeClaim: pvc-1

   where,

   target

   Indicates the volume target on which the operation is performed.

   persistentVolumeClaim

   Name of the PersistentVolumeClaim.

   backOfflimit

   Specifies the maximum number of retries before marking the reclaim space operation as failed. The default value is 6. The allowed maximum and minimum values are 60 and 0 respectively.

   retryDeadlineSeconds

   Specifies the duration in which the operation might retire in seconds and it is relative to the start time. The value must be a positive integer. The default value is 600 seconds and the allowed maximum value is 1800 seconds.

2. Delete the custom resource after completion of the operation.

Procedure

1. Create and apply the following custom resource for reclaim space operation

   apiVersion: csiaddons.openshift.io/v1alpha1
   kind: ReclaimSpaceCronJob
   metadata:
     name: reclaimspacecronjob-sample
   spec:
     jobTemplate:
       spec:
         target:
           persistentVolumeClaim: data-pvc
     schedule: '@weekly'
     concurrencyPolicy: Forbid

   where,

   concurrencyPolicy

   Describes the changes when a new ReclaimSpaceJob is scheduled by the ReclaimSpaceCronJob, while a previous ReclaimSpaceJob is still running. The default Forbid prevents starting a new job whereas Replace can be used to delete the running job potentially in a failure state and create a new one.

   failedJobsHistoryLimit

   Specifies the number of failed ReclaimSpaceJobs that are kept for troubleshooting.

   jobTemplate

   Specifies the ReclaimSpaceJob.spec structure that describes the details of the requested ReclaimSpaceJob operation.

   successfulJobsHistoryLimit

   Specifies the number of successful ReclaimSpaceJob operations.

   schedule

   Specifieds the and/or interval of the recurring operation request and it is in the same format as the Kubernetes CronJobs.

2. Delete the ReclaimSpaceCronJob custom resource when execution of reclaim space operation is no longer needed or when the target PVC is deleted.

Procedure

1. Click Storage → Persistent Volume Claims from the OpenShift Web Console.

2. To create a clone, do one of the following:

   • Beside the desired PVC, click Action menu (⋮) → Clone PVC.
   • Click on the PVC that you want to clone and click Actions → Clone PVC.
3. Enter a Name for the clone.

4. Select the access mode of your choice.

   Important

   The ReadOnlyMany (ROX) access mode is a Developer Preview feature and is subject to Developer Preview support limitations. Developer Preview releases are not intended to be run in production environments and are not supported through the Red Hat Customer Portal case management system. If you need assistance with ReadOnlyMany feature, reach out to the ocs-devpreview@redhat.com mailing list and a member of the Red Hat Development Team will assist you as quickly as possible based on availability and work schedules. See Creating a clone or restoring a snapshot with the new readonly access mode to use the ROX access mode.

5. Click Clone. You are redirected to the new PVC details page.

6. Wait for the cloned PVC status to become Bound.

   The cloned PVC is now available to be consumed by the pods. This cloned PVC is independent of its dataSource PVC.

Procedure

1. In the OpenShift Web Console, click Storage → Persistent Volume Claims
2. Set the Project to openshift-storage.

3. Click Create PersistentVolumeClaim.

   1. Specify Storage Class, ocs-storagecluster-ceph-nfs.
   2. Specify the PVC Name, for example, my-nfs-export.
   3. Select the required Access Mode.
   4. Specify a Size as per application requirement.

   5. Select Volume mode as Filesystem.

      Note: Block mode is not supported for NFS PVCs

   6. Click Create and wait until the PVC is in Bound status.

Procedure

1. On the OpenShift Container Platform web console, navigate to Workloads → Pods.
2. Click Create Pod to create a new application pod.
3. Under the metadata section add a name. For example, nfs-export-example, with namespace as openshift-storage.

4. Under the spec: section, add containers: section with image and volumeMounts sections:

   apiVersion: v1
   kind: Pod
   metadata:
    name: nfs-export-example
    namespace: openshift-storage
   spec:
    containers:
      - name: web-server
        image: nginx
        volumeMounts:
          - name: <volume_name>
            mountPath: /var/lib/www/html

   For example:

   apiVersion: v1
   kind: Pod
   metadata:
    name: nfs-export-example
    namespace: openshift-storage
   spec:
    containers:
      - name: web-server
        image: nginx
        volumeMounts:
          - name: nfs-export-pvc
            mountPath: /var/lib/www/html

5. Under the spec: section, add volumes: section to add the NFS PVC as a volume for the application pod:

   volumes:
     - name: <volume_name>
       persistentVolumeClaim:
         claimName: <pvc_name>

   For example:

   volumes:
     - name: nfs-export-pvc
       persistentVolumeClaim:
         claimName: my-nfs-export

Procedure

1. After the nfs flag is enabled, singe-server CephNFS is deployed by Rook. You need to fetch the value of the ceph_nfs field for the nfs-ganesha server to use in the next step:

   $ oc get pods -n openshift-storage | grep rook-ceph-nfs

   $ oc describe pod  <name of the rook-ceph-nfs pod> | grep ceph_nfs

   For example:

   $ oc describe pod rook-ceph-nfs-ocs-storagecluster-cephnfs-a-7bb484b4bf-bbdhs | grep ceph_nfs
     ceph_nfs=my-nfs

2. Expose the NFS server outside of the OpenShift cluster by creating a Kubernetes LoadBalancer Service. The example below creates a LoadBalancer Service and references the NFS server created by OpenShift Data Foundation.

   apiVersion: v1
   kind: Service
   metadata:
    name: rook-ceph-nfs-ocs-storagecluster-cephnfs-load-balancer
    namespace: openshift-storage
   spec:
    ports:
      - name: nfs
        port: 2049
    type: LoadBalancer
    externalTrafficPolicy: Local
    selector:
      app: rook-ceph-nfs
      ceph_nfs: <my-nfs>
      instance: a

   Replace <my-nfs> with the value you got in step 1.

3. Collect connection information. The information external clients need to connect to an export comes from the Persistent Volume (PV) created for the PVC, and the status of the LoadBalancer Service created in the previous step.

   1. Get the share path from the PV.

      1. Get the name of the PV associated with the NFS export’s PVC:

         $ oc get pvc <pvc_name> --output jsonpath='{.spec.volumeName}'
         pvc-39c5c467-d9d3-4898-84f7-936ea52fd99d

         Replace <pvc_name> with your own PVC name. For example:

         oc get pvc pvc-39c5c467-d9d3-4898-84f7-936ea52fd99d --output jsonpath='{.spec.volumeName}'
         pvc-39c5c467-d9d3-4898-84f7-936ea52fd99d

      2. Use the PV name obtained previously to get the NFS export’s share path:

         $ oc get pv pvc-39c5c467-d9d3-4898-84f7-936ea52fd99d --output jsonpath='{.spec.csi.volumeAttributes.share}'
         /0001-0011-openshift-storage-0000000000000001-ba9426ab-d61b-11ec-9ffd-0a580a800215

   2. Get an ingress address for the NFS server. A service’s ingress status may have multiple addresses. Choose the one desired to use for external clients. In the example below, there is only a single address: the host name ingress-id.somedomain.com.

      $ oc -n openshift-storage get service rook-ceph-nfs-ocs-storagecluster-cephnfs-load-balancer --output jsonpath='{.status.loadBalancer.ingress}'
      [{"hostname":"ingress-id.somedomain.com"}]

4. Connect the external client using the share path and ingress address from the previous steps. The following example mounts the export to the client’s directory path /export/mount/path:

   $ mount -t nfs4 -o proto=tcp ingress-id.somedomain.com:/0001-0011-openshift-storage-0000000000000001-ba9426ab-d61b-11ec-9ffd-0a580a800215 /export/mount/path

   If this does not work immediately, it could be that the Kubernetes environment is still taking time to configure the network resources to allow ingress to the NFS server.