Chapter 2. Storage classes

Procedure

1. In the OpenShift Web Console, navigate to Storage StorageClasses.
2. Click Create Storage Class.
3. Enter the storage class Name and Description.
4. Select either Delete or Retain for the Reclaim Policy. By default, Delete is selected.
5. Select either Immediate or WaitForFirstConsumer as the Volume binding mode. WaitForConsumer is set as the default option.
6. Select RBD Provisioner openshift-storage.rbd.csi.ceph.com which is the plugin used for provisioning the persistent volumes.
7. Select Storage Pool where the volume data is stored from the list or create a new pool.

8. Select the Enable encryption checkbox. There are two options available to set the KMS connection details:

   • Select existing KMS connection: Select an existing KMS connection from the drop-down list. The list is populated from the the connection details available in the csi-kms-connection-details ConfigMap.

     1. Select the Provider from the drop down.
     2. Select the Key service for the given provider from the list.

   • Create new KMS connection: This is applicable for vaulttokens and Thales CipherTrust Manager (using KMIP) only.

     1. Select the Key Management Service Provider.

     2. If Vault is selected as the Key Management Service Provider, follow these steps:

        1. Enter a unique Connection Name, host Address of the Vault server ('https://<hostname or ip>'), Port number and Token.

        2. Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:

           1. Enter the Key Value secret path in Backend Path that is dedicated and unique to OpenShift Data Foundation.
           2. Optional: Enter TLS Server Name and Vault Enterprise Namespace.
           3. Upload the respective PEM encoded certificate file to provide the CA Certificate, Client Certificate and Client Private Key .
           4. Click Save.

     3. If Thales CipherTrust Manager (using KMIP) is selected as the Key Management Service Provider, follow these steps:

        1. Enter a unique Connection Name.
        2. In the Address and Port sections, enter the IP of Thales CipherTrust Manager and the port where the KMIP interface is enabled. For example, Address: 123.34.3.2, Port: 5696.
        3. Upload the Client Certificate, CA certificate, and Client Private Key.
        4. Enter the Unique Identifier for the key to be used for encryption and decryption, generated above.
        5. The TLS Server field is optional and used when there is no DNS entry for the KMIP endpoint. For example, kmip_all_<port>.ciphertrustmanager.local.
     4. Click Save.
     5. Click Create.

9. Edit the ConfigMap to add the vaultBackend parameter if the HashiCorp Vault setup does not allow automatic detection of the Key/Value (KV) secret engine API version used by the backend path.

   Note

   vaultBackend is an optional parameters that is added to the configmap to specify the version of the KV secret engine API associated with the backend path. Ensure that the value matches the KV secret engine API version that is set for the backend path, otherwise it might result in a failure during persistent volume claim (PVC) creation.

   1. Identify the encryptionKMSID being used by the newly created storage class.

      1. On the OpenShift Web Console, navigate to Storage Storage Classes.
      2. Click the Storage class name YAML tab.

      3. Capture the encryptionKMSID being used by the storage class.

         Example:

         encryptionKMSID: 1-vault

   2. On the OpenShift Web Console, navigate to Workloads ConfigMaps.
   3. To view the KMS connection details, click csi-kms-connection-details.

   4. Edit the ConfigMap.

      1. Click Action menu (⋮) Edit ConfigMap.

      2. Add the vaultBackend parameter depending on the backend that is configured for the previously identified encryptionKMSID.

         You can assign kv for KV secret engine API, version 1 and kv-v2 for KV secret engine API, version 2.

         Example:

          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: csi-kms-connection-details
          [...]
          data:
            1-vault: |-
              {
                "encryptionKMSType": "vaulttokens",
                "kmsServiceName": "1-vault",
                [...]
                "vaultBackend": "kv-v2"
              }
            2-vault: |-
              {
                "encryptionKMSType": "vaulttenantsa",
                [...]
                "vaultBackend": "kv"
              }

      3. Click Save

1. If you know which failure domain the failing OSD is in, run the following command to get the exact replica1-pool-name required for the next steps. If you do not know where the failing OSD is, skip to step 2.

   $ oc get cephblockpools

   Example output:

   NAME                                          PHASE
   ocs-storagecluster-cephblockpool              Ready
   ocs-storagecluster-cephblockpool-us-south-1   Ready
   ocs-storagecluster-cephblockpool-us-south-2   Ready
   ocs-storagecluster-cephblockpool-us-south-3   Ready

   Copy the corresponding failure domain name for use in next steps, then skip to step 4.

2. Find the OSD pod that is in Error state or CrashLoopBackoff state to find the failing OSD:

   $ oc get pods -nopenshift-storage -l app=rook-ceph-osd  | grep 'CrashLoopBackOff\|Error'

3. Identify the replica-1 pool that had the failed OSD.

   1. Identify the node where the failed OSD was running:

      failed_osd_id=0 #replace with the ID of the failed OSD

   2. Identify the failureDomainLabel for the node where the failed OSD was running:

      failure_domain_label=$(oc get storageclass ocs-storagecluster-ceph-non-resilient-rbd -o yaml | grep domainLabel |head -1 |awk -F':' '{print $2}')

      failure_domain_value=$”(oc get pods $failed_osd_id -oyaml |grep topology-location-zone |awk ‘{print $2}’)”

      The output shows the replica-1 pool name whose OSD is failing, for example:

      replica1-pool-name= "ocs-storagecluster-cephblockpool-$failure_domain_value”

      where $failure_domain_value is the failureDomainName.

4. Delete the replica-1 pool.

   1. Connect to the toolbox pod:

      toolbox=$(oc get pod -l app=rook-ceph-tools -n openshift-storage -o jsonpath='{.items[*].metadata.name}')
      
      oc rsh $toolbox -n openshift-storage

   2. Delete the replica-1 pool. Note that you have to enter the replica-1 pool name twice in the command, for example:

      ceph osd pool rm <replica1-pool-name> <replica1-pool-name> --yes-i-really-really-mean-it

      Replace replica1-pool-name with the failure domain name identified earlier.

5. Purge the failing OSD by following the steps in section "Replacing operational or failed storage devices" based on your platform in the Replacing devices guide.

6. Restart the rook-ceph operator:

   $ oc delete pod -l rook-ceph-operator -nopenshift-storage

7. Recreate any affected applications in that avaialbity zone to start using the new pool with same name.