Chapter 9. Troubleshooting CephFS PVC creation in external mode

Check for CephFS pvc stuck in Pending status.

# oc get pvc -n <namespace>

Example output :

NAME                      STATUS    VOLUME
CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
ngx-fs-pxknkcix20-pod     Pending
                          ocs-external-storagecluster-cephfs  28h
[...]

Check the describe output to see the events for respective pvc.

Expected error message is cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

# oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

Example output:

Name:          ngx-fs-pxknkcix20-pod
Namespace:     nginx-file
StorageClass:  ocs-external-storagecluster-cephfs
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
Events:
 Type     Reason              Age                   From                                                                                                                      Message
 ----     ------              ----                  ----                                                                                                                      -------
 Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
 (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
 running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
 csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
 cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)

Check the settings for the <cephfs metadata pool name> (here cephfs_metadata ) and <cephfs data pool name> (here cephfs_data). For running the command, you will need jq preinstalled in the Red Hat Ceph Storage client node.

# ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
{
  "cephfs": {}
}
"cephfs_metadata"
{
   "cephfs": {}
}

Set the application type for CephFS pool.

Verify if the settings are applied.

 # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
  {
    "cephfs": {
      "data": "cephfs"
   }
  }
  "cephfs_metadata"
  {
    "cephfs": {
      "metadata": "cephfs"
    }
  }

Check the CephFS PVC status again. The PVC should now be in Bound state.

# oc get pvc -n <namespace>

Example output :

NAME                      STATUS    VOLUME
CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
1Mi       RWO             ocs-external-storagecluster-cephfs  29h
[...]