Troubleshooting OpenShift Data Foundation

Procedure

1. Specify a blank node selector for the openshift-storage namespace.

   $ oc annotate namespace openshift-storage openshift.io/node-selector=

2. Delete the original pods generated by the DaemonSets.

   oc delete pod -l app=csi-cephfsplugin -n openshift-storage
   oc delete pod -l app=csi-rbdplugin -n openshift-storage

Procedure

1. Log in to OpenShift Container Platform Web Console.
2. Click Workloads → Secrets

3. To update the ocs-kms-token used for cluster wide encryption:

   1. Set the Project to openshift-storage.
   2. Click ocs-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.
   4. Click Save.

4. To update the ceph-csi-kms-token for a given project or namespace with encrypted persistent volumes:

   1. Select the required Project.
   2. Click ceph-csi-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.

   4. Click Save.

      Note

      The token can be deleted only after all the encrypted PVCs using the ceph-csi-kms-token have been deleted.

1. Check the storage class associated with OpenShift Data Foundation cluster’s PVCs with the following command:

   $ oc get pvc -n openshift-storage

2. Check the output. For clusters with Local Storage Operator, the PVCs associated with ocs-deviceset use the storage class localblock. The output looks similar to the following:

   NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   db-noobaa-db-0            Bound    pvc-d96c747b-2ab5-47e2-b07e-1079623748d8   50Gi       RWO            ocs-storagecluster-ceph-rbd   114s
   ocs-deviceset-0-0-lzfrd   Bound    local-pv-7e70c77c                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-1-0-7rggl   Bound    local-pv-b19b3d48                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-2-0-znhk8   Bound    local-pv-e9f22cdc                          1769Gi     RWO            localblock                    2m10s

1. Check if the openshift-storage namespace is stuck in Terminating state upon deletion.

   $ oc get project -n <namespace>

   Output:

   NAME                DISPLAY NAME   STATUS
   openshift-storage                  Terminating

2. Check for the NamespaceFinalizersRemaining and NamespaceContentRemaining messages in the STATUS section of the command output and perform the next step for each of the listed resources.

   $ oc get project openshift-storage -o yaml

   Example output :

   status:
     conditions:
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All resources successfully discovered
       reason: ResourcesDiscovered
       status: "False"
       type: NamespaceDeletionDiscoveryFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All legacy kube types successfully parsed
       reason: ParsedGroupVersions
       status: "False"
       type: NamespaceDeletionGroupVersionParsingFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All content successfully deleted, may be waiting on finalization
       reason: ContentDeleted
       status: "False"
       type: NamespaceDeletionContentFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some resources are remaining: cephobjectstoreusers.ceph.rook.io has
         1 resource instances'
       reason: SomeResourcesRemain
       status: "True"
       type: NamespaceContentRemaining
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io
         in 1 resource instances'
       reason: SomeFinalizersRemain
       status: "True"
       type: NamespaceFinalizersRemaining

3. Delete all the remaining resources listed in the previous step.

   For each of the resources to be deleted, do the following:

   1. Get the object kind of the resource which needs to be removed. See the message in the above output.

      Example :

      message: Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io

      Here cephobjectstoreuser.ceph.rook.io is the object kind.

   2. Get the Object name corresponding to the object kind.

      $ oc get  <Object-kind> -n  <project-name>

      Example :

      $ oc get cephobjectstoreusers.ceph.rook.io -n openshift-storage

      Example output:

      NAME                           AGE
      noobaa-ceph-objectstore-user   26h

   3. Patch the resources.

      $ oc patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

      Example:

      $ oc patch -n openshift-storage cephobjectstoreusers.ceph.rook.io/noobaa-ceph-objectstore-user \
      --type=merge -p '{"metadata": {"finalizers":null}}'

      Output:

      cephobjectstoreuser.ceph.rook.io/noobaa-ceph-objectstore-user patched

4. Verify that the openshift-storage project is deleted.

   $ oc get project openshift-storage

   Output:

   Error from server (NotFound): namespaces "openshift-storage" not found

   If the issue persists, reach out to Red Hat Support.

1. Check for CephFS pvc stuck in Pending status.

   # oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Pending
                             ocs-external-storagecluster-cephfs  28h
   [...]

2. Check the describe output to see the events for respective pvc.

   Expected error message is cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

   # oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

   Example output:

   Name:          ngx-fs-pxknkcix20-pod
   Namespace:     nginx-file
   StorageClass:  ocs-external-storagecluster-cephfs
   Status:        Pending
   Volume:
   Labels:        <none>
   Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:
   Access Modes:
   VolumeMode:    Filesystem
   Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
   Events:
    Type     Reason              Age                   From                                                                                                                      Message
    ----     ------              ----                  ----                                                                                                                      -------
    Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
    (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
    running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
    csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
    cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)

3. Check the settings for the <cephfs metadata pool name> (here cephfs_metadata ) and <cephfs data pool name> (here cephfs_data). For running the command, you will need jq preinstalled in the Red Hat Ceph Storage client node.

   # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
   {
     "cephfs": {}
   }
   "cephfs_metadata"
   {
      "cephfs": {}
   }

4. Set the application type for CephFS pool.

   • Run the following commands on the Red Hat Ceph Storage client node :

     # ceph osd pool application set <cephfs metadata pool name> cephfs metadata cephfs

     # ceph osd pool application set <cephfs data pool name> cephfs data cephfs

5. Verify if the settings are applied.

    # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
     {
       "cephfs": {
         "data": "cephfs"
      }
     }
     "cephfs_metadata"
     {
       "cephfs": {
         "metadata": "cephfs"
       }
     }

6. Check the CephFS PVC status again. The PVC should now be in Bound state.

   # oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
   1Mi       RWO             ocs-external-storagecluster-cephfs  29h
   [...]

Procedure

1. Scale down the rook-ceph-operator and ocs operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   # oc scale deployment ocs-operator --replicas=0 -n openshift-storage

2. Create a backup of all deployments in openshift-storage namespace.

   # mkdir backup

   # cd backup

   # oc project openshift-storage

   # for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

3. Patch the OSD deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep.

   # for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

4. Retrieve the monstore cluster map from all the OSDs.

   1. Create the recover_mon.sh script.

      #!/bin/bash
      ms=/tmp/monstore
      
      rm -rf $ms
      mkdir $ms
      
      for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do
      
        echo "Starting with pod: $osd_pod"
      
        podname=$(echo $osd_pod|sed 's/pod\///g')
        oc exec $osd_pod -- rm -rf $ms
        oc cp $ms $podname:$ms
      
        rm -rf $ms
        mkdir $ms
      
        echo "pod in loop: $osd_pod ; done deleting local dirs"
      
        oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
        echo "Done with COT on pod: $osd_pod"
      
        oc cp $podname:$ms $ms
      
        echo "Finished pulling COT data from pod: $osd_pod"
      done

   2. Run the recover_mon.sh script.

      # chmod +x recover_mon.sh

      # ./recover_mon.sh

5. Patch the MON deployments, and run it with the command parameter as sleep.

   1. Edit the MON deployments.

      # for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

   2. Patch the MON deployments to increase the initialDelaySeconds.

      # oc get deployment rook-ceph-mon-a -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      # oc get deployment rook-ceph-mon-b -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      # oc get deployment rook-ceph-mon-c -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

6. Copy the previously retrieved monstore to the mon-a pod.

   # oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

7. Navigate into the MON pod and change the ownership of the retrieved monstore.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # chown -R ceph:ceph /tmp/monstore

8. Copy the keyring template file before rebuilding the mon db.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # cp /etc/ceph/keyring-store/keyring /tmp/keyring

   # cat /tmp/keyring
     [mon.]
       key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
       caps mon = "allow *"
     [client.admin]
       key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
       auid = 0
       caps mds = "allow *"
       caps mgr = "allow *"
       caps mon = "allow *"
       caps osd = "allow *”

9. Identify the keyring of all other Ceph daemons (MGR, MDS, RGW, Crash, CSI and CSI provisioners) from its respective secrets.

   # oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d
   
   [mds.ocs-storagecluster-cephfilesystem-a]
   key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
   caps mon = "allow profile mds"
   caps osd = "allow *"
   caps mds = "allow"

   Example keyring file, /etc/ceph/ceph.client.admin.keyring:

   [mon.]
   	key = AQDxTF1hNgLTNxAAi51cCojs01b4I5E6v2H8Uw==
   	caps mon = "allow "
   [client.admin]
           key = AQDxTF1hpzguOxAA0sS8nN4udoO35OEbt3bqMQ==
           caps mds = "allow " caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mds.ocs-storagecluster-cephfilesystem-a] key = AQCKTV1horgjARAA8aF/BDh/4+eG4RCNBCl+aw== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [mds.ocs-storagecluster-cephfilesystem-b] key = AQCKTV1hN4gKLBAA5emIVq3ncV7AMEM1c1RmGA== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [client.rgw.ocs.storagecluster.cephobjectstore.a] key = AQCOkdBixmpiAxAA4X7zjn6SGTI9c1MBflszYA== caps mon = "allow rw" caps osd = "allow rwx" [mgr.a] key = AQBOTV1hGYOEORAA87471+eIZLZtptfkcHvTRg== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [client.crash] key = AQBOTV1htO1aGRAAe2MPYcGdiAT+Oo4CNPSF1g== caps mgr = "allow rw" caps mon = "allow profile crash" [client.csi-cephfs-node] key = AQBOTV1hiAtuBBAAaPPBVgh1AqZJlDeHWdoFLw== caps mds = "allow rw" caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs *=" [client.csi-cephfs-provisioner] key = AQBNTV1hHu6wMBAAzNXZv36aZJuE1iz7S7GfeQ== caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs metadata="
   [client.csi-rbd-node]
   	key = AQBNTV1h+LnkIRAAWnpIN9bUAmSHOvJ0EJXHRw==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"
   [client.csi-rbd-provisioner]
   	key = AQBNTV1hMNcsExAAvA3gHB2qaY33LOdWCvHG/A==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"

   Important
   • For client.csi related keyring, refer to the previous keyring file output and add the default caps after fetching the key from its respective OpenShift Data Foundation secret.
   • OSD keyring is added automatically post recovery.

10. Navigate into the mon-a pod, and verify that the monstore has monmap.

    1. Navigate into the mon-a pod.

       # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

    2. Verify that the monstore has monmap.

       # ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

       # monmaptool /tmp/monmap --print

11. Optional: If the monmap is missing then create a new monmap.

    # monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>

    <mon-a-id>

    Is the ID of the mon-a pod.

    <mon-a-ip>

    Is the IP address of the mon-a pod.

    <mon-b-id>

    Is the ID of the mon-b pod.

    <mon-b-ip>

    Is the IP address of the mon-b pod.

    <mon-c-id>

    Is the ID of the mon-c pod.

    <mon-c-ip>

    Is the IP address of the mon-c pod.

    <fsid>

    Is the file system ID.

12. Verify the monmap.

    # monmaptool /root/monmap --print

13. Import the monmap.

    Important

    Use the previously created keyring file.

    # ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

    # chown -R ceph:ceph /tmp/monstore

14. Create a backup of the old store.db file.

    # mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

15. Copy the rebuild store.db file to the monstore directory.

    # mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

16. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods.

    # oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

    # oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

    <id>

    Is the ID of the MON pod

17. Navigate into the rest of the MON pods and change the ownership of the copied monstore.

    # oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

    <id>

    Is the ID of the MON pod

18. Revert the patched changes.

    • For MON deployments:

      # oc replace --force -f <mon-deployment.yaml>

      <mon-deployment.yaml>

      Is the MON deployment yaml file

    • For OSD deployments:

      # oc replace --force -f <osd-deployment.yaml>

      <osd-deployment.yaml>

      Is the OSD deployment yaml file

    • For MGR deployments:

      # oc replace --force -f <mgr-deployment.yaml>

      <mgr-deployment.yaml>

      Is the MGR deployment yaml file

      Important

      Ensure that the MON, MGR and OSD pods are up and running.

19. Scale up the rook-ceph-operator and ocs-operator deployments.

    # oc -n openshift-storage scale deployment ocs-operator --replicas=1

Verification steps

1. Check the Ceph status to confirm that CephFS is running.

   # ceph -s

   Example output:

   cluster:
      id:     f111402f-84d1-4e06-9fdb-c27607676e55
      health: HEALTH_ERR
               1 filesystem is offline
               1 filesystem is online with fewer MDS than max_mds
               3 daemons have recently crashed
   
      services:
        mon: 3 daemons, quorum b,c,a (age 15m)
        mgr: a(active, since 14m)
        mds: ocs-storagecluster-cephfilesystem:0
        osd: 3 osds: 3 up (since 15m), 3 in (since 2h)
   
      data:
        pools:   3 pools, 96 pgs
        objects: 500 objects, 1.1 GiB
        usage:   5.5 GiB used, 295 GiB / 300 GiB avail
        pgs:     96 active+clean

   Important

   If the filesystem is offline or MDS service is missing, you need to restore the CephFS. For more information, see Section 10.1, “Restoring the CephFS”.

2. Check the Multicloud Object Gateway (MCG) status. It should be active, and the backingstore and bucketclass should be in Ready state.

   noobaa status -n openshift-storage

   Important

   If the MCG is not in the active state, and the backingstore and bucketclass not in the Ready state, you need to restart all the MCG related pods. For more information, see Section 10.2, “Restoring the Multicloud Object Gateway”.