Troubleshooting OpenShift Container Storage

Procedure

1. Specify a blank node selector for the openshift-storage namespace.

   $ oc annotate namespace openshift-storage openshift.io/node-selector=

2. Delete the original pods generated by the DaemonSets.

   oc delete pod -l app=csi-cephfsplugin -n openshift-storage
   oc delete pod -l app=csi-rbdplugin -n openshift-storage

Procedure

1. Log in to OpenShift Container Platform Web Console.
2. Click Workloads → Secrets

3. To update the ocs-kms-token used for cluster wide encryption:

   1. Set the Project to openshift-storage.
   2. Click ocs-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.
   4. Click Save.

4. To update the ceph-csi-kms-token for a given project or namespace with encrypted persistent volumes:

   1. Select the required Project.
   2. Click ceph-csi-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.

   4. Click Save.

      Note

      The token can be deleted only after all the encrypted PVCs using the ceph-csi-kms-token have been deleted.

1. Check the storage class associated with OpenShift Container Storage cluster’s PVCs with the following command:

   $ oc get pvc -n openshift-storage

2. Check the output. For clusters with Local Storage Operator, the PVCs associated with ocs-deviceset use the storage class localblock. The output looks similar to the following:

   NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   db-noobaa-db-0            Bound    pvc-d96c747b-2ab5-47e2-b07e-1079623748d8   50Gi       RWO            ocs-storagecluster-ceph-rbd   114s
   ocs-deviceset-0-0-lzfrd   Bound    local-pv-7e70c77c                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-1-0-7rggl   Bound    local-pv-b19b3d48                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-2-0-znhk8   Bound    local-pv-e9f22cdc                          1769Gi     RWO            localblock                    2m10s

1. Check if the openshift-storage namespace is stuck in Terminating state upon deletion.

   $ oc get project -n <namespace>

   Output:

   NAME                DISPLAY NAME   STATUS
   openshift-storage                  Terminating

2. Check for the NamespaceFinalizersRemaining and NamespaceContentRemaining messages in the STATUS section of the command output and perform the next step for each of the listed resources.

   $ oc get project openshift-storage -o yaml

   Example output :

   status:
     conditions:
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All resources successfully discovered
       reason: ResourcesDiscovered
       status: "False"
       type: NamespaceDeletionDiscoveryFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All legacy kube types successfully parsed
       reason: ParsedGroupVersions
       status: "False"
       type: NamespaceDeletionGroupVersionParsingFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All content successfully deleted, may be waiting on finalization
       reason: ContentDeleted
       status: "False"
       type: NamespaceDeletionContentFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some resources are remaining: cephobjectstoreusers.ceph.rook.io has
         1 resource instances'
       reason: SomeResourcesRemain
       status: "True"
       type: NamespaceContentRemaining
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io
         in 1 resource instances'
       reason: SomeFinalizersRemain
       status: "True"
       type: NamespaceFinalizersRemaining

3. Delete all the remaining resources listed in the previous step.

   For each of the resources to be deleted, do the following:

   1. Get the object kind of the resource which needs to be removed. See the message in the above output.

      Example :

      message: Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io

      Here cephobjectstoreuser.ceph.rook.io is the object kind.

   2. Get the Object name corresponding to the object kind.

      $ oc get  <Object-kind> -n  <project-name>

      Example :

      $ oc get cephobjectstoreusers.ceph.rook.io -n openshift-storage

      Example output:

      NAME                           AGE
      noobaa-ceph-objectstore-user   26h

   3. Patch the resources.

      $ oc patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

      Example:

      $ oc patch -n openshift-storage cephobjectstoreusers.ceph.rook.io/noobaa-ceph-objectstore-user \
      --type=merge -p '{"metadata": {"finalizers":null}}'

      Output:

      cephobjectstoreuser.ceph.rook.io/noobaa-ceph-objectstore-user patched

4. Verify that the openshift-storage project is deleted.

   $ oc get project  openshift-storage

   Output:

   Error from server (NotFound): namespaces "openshift-storage" not found

   If the issue persists, reach out to Red Hat Support.

1. Check for CephFS pvc stuck in Pending status.

   # oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Pending
                             ocs-external-storagecluster-cephfs  28h
   [...]

2. Check the describe output to see the events for respective pvc.

   Expected error message is cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

   # oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

   Example output:

   Name:          ngx-fs-pxknkcix20-pod
   Namespace:     nginx-file
   StorageClass:  ocs-external-storagecluster-cephfs
   Status:        Pending
   Volume:
   Labels:        <none>
   Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:
   Access Modes:
   VolumeMode:    Filesystem
   Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
   Events:
    Type     Reason              Age                   From                                                                                                                      Message
    ----     ------              ----                  ----                                                                                                                      -------
    Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
    (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
    running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
    csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
    cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)

3. Check the settings for the <cephfs metadata pool name> (here cephfs_metadata ) and <cephfs data pool name> (here cephfs_data). For running the command, you will need jq preinstalled in the Red Hat Ceph Storage client node.

   # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
   {
     "cephfs": {}
   }
   "cephfs_metadata"
   {
      "cephfs": {}
   }

4. Set the application type for CephFS pool.

   • Run the following commands on the Red Hat Ceph Storage client node :

     # ceph osd pool application set <cephfs metadata pool name> cephfs metadata cephfs

     # ceph osd pool application set <cephfs data pool name> cephfs data cephfs

5. Verify if the settings are applied.

    # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
     {
       "cephfs": {
         "data": "cephfs"
      }
     }
     "cephfs_metadata"
     {
       "cephfs": {
         "metadata": "cephfs"
       }
     }

6. Check the CephFS PVC status again. The PVC should now be in Bound state.

   # oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
   1Mi       RWO             ocs-external-storagecluster-cephfs  29h
   [...]

Procedure

1. Scale down the rook-ceph-operator and ocs operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   # oc scale deployment ocs-operator --replicas=0 -n openshift-storage

2. Create a backup of all deployments in the openshift-storage namespace.

   # mkdir backup

   # cd backup

   # oc project openshift-storage

   # for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

3. Patch the OSD deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep.

   # for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

4. Retrieve the monstore cluster map from all the OSDs.

   1. Create the recover_mon.sh script.

      #!/bin/bash
      ms=/tmp/monstore
      
      rm -rf $ms
      mkdir $ms
      
      for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do
      
        echo "Starting with pod: $osd_pod"
      
        podname=$(echo $osd_pod|sed 's/pod\///g')
        oc exec $osd_pod -- rm -rf $ms
        oc cp $ms $podname:$ms
      
        rm -rf $ms
        mkdir $ms
      
        echo "pod in loop: $osd_pod ; done deleting local dirs"
      
        oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
        echo "Done with COT on pod: $osd_pod"
      
        oc cp $podname:$ms $ms
      
        echo "Finished pulling COT data from pod: $osd_pod"
      done

   2. Run the recover_mon.sh script.

      # chmod +x recover_mon.sh

      # ./recover_mon.sh

5. Patch the MON deployments, and run it with the command parameter as sleep.

   1. Edit the MON deployments.

      # for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

   2. Patch the MON deployments to increase the initialDelaySeconds.

      # oc get deployment rook-ceph-mon-a -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      # oc get deployment rook-ceph-mon-b -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      # oc get deployment rook-ceph-mon-c -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

6. Copy the previously retrieved monstore to the mon-a pod.

   # oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

7. Navigate into the MON pod and change the ownership of the retrieved monstore.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # chown -R ceph:ceph /tmp/monstore

8. Set the appropriate capabilities before rebuilding the MON DB.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # cp /etc/ceph/keyring-store/keyring /tmp/keyring

   # cat /tmp/keyring
     [mon.]
       key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
       caps mon = "allow *"
     [client.admin]
       key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
       auid = 0
       caps mds = "allow *"
       caps mgr = "allow *"
       caps mon = "allow *"
       caps osd = "allow *”

9. Identify the keyring of all the other Ceph daemons (MGR, MDS, RGW, Crash, CSI and CSI provisioners) from its respective secrets.

   # oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d
   
   [mds.ocs-storagecluster-cephfilesystem-a]
   key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
   caps mon = "allow profile mds"
   caps osd = "allow *"
   caps mds = "allow"

   Example keyring file, /etc/ceph/ceph.client.admin.keyring:

   [mon.]
   	key = AQDxTF1hNgLTNxAAi51cCojs01b4I5E6v2H8Uw==
   	caps mon = "allow " [mds.ocs-storagecluster-cephfilesystem-a] key = AQCKTV1horgjARAA8aF/BDh/4+eG4RCNBCl+aw== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [mds.ocs-storagecluster-cephfilesystem-b] key = AQCKTV1hN4gKLBAA5emIVq3ncV7AMEM1c1RmGA== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [client.admin] key = AQDxTF1hpzguOxAA0sS8nN4udoO35OEbt3bqMQ== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.crash] key = AQBOTV1htO1aGRAAe2MPYcGdiAT+Oo4CNPSF1g== caps mgr = "allow rw" caps mon = "allow profile crash" [client.csi-cephfs-node] key = AQBOTV1hiAtuBBAAaPPBVgh1AqZJlDeHWdoFLw== caps mds = "allow rw" caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs *="
   [client.csi-cephfs-provisioner]
   	key = AQBNTV1hHu6wMBAAzNXZv36aZJuE1iz7S7GfeQ==
   	caps mgr = "allow rw"
   	caps mon = "allow r"
   	caps osd = "allow rw tag cephfs metadata=*"
   [client.csi-rbd-node]
   	key = AQBNTV1h+LnkIRAAWnpIN9bUAmSHOvJ0EJXHRw==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"
   [client.csi-rbd-provisioner]
   	key = AQBNTV1hMNcsExAAvA3gHB2qaY33LOdWCvHG/A==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"
   [mgr.a]
   	key = AQBOTV1hGYOEORAA87471+eIZLZtptfkcHvTRg==
   	caps mds = "allow *"
   	caps mon = "allow profile mgr"
   	caps osd = "allow *"

   Important
   • For client.csi related keyring, add the default caps after fetching the key from its respective OpenShift Container Storage secret.
   • OSD keyring is added automatically post recovery.

10. Navigate into the mon-a pod.

    Verify that the monstore has monmap.

    # ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

    # monmaptool /tmp/monmap --print

    If the monmap is missing then create a new monmap.

    # monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>

    <mon-a-id>

    Is the ID of the mon-a pod

    <mon-a-ip>

    Is the IP address of the mon-a pod

    <mon-b-id>

    Is the ID of the mon-b pod

    <mon-b-ip>

    Is the IP address of the mon-b pod

    <mon-c-id>

    Is the ID of the mon-c pod

    <mon-c-ip>

    Is the IP address of the mon-c pod

    <fsid>

    Is the file system ID

11. Verify the monmap.

    # monmaptool /root/monmap --print

12. Import the monmap.

    Important

    Use the previously created keyring file.

    # ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

    # chown -R ceph:ceph /tmp/monstore

13. Create a backup of the old store.db file.

    # mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

14. Copy the rebuild store.db file to the monstore directory.

    # mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

15. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods.

    # oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

    # oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

    <id>

    Is the ID of the MON pod

16. Navigate into the rest of the MON pods and change the ownership of the copied monstore.

    # oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

    <id>

    Is the ID of the MON pod

17. Revert the patched changes.

    • For MON deployments:

      # oc replace --force -f <mon-deployment.yaml>

      <mon-deployment.yaml>

      Is the MON deployment yaml file

    • For OSD deployments:

      # oc replace --force -f <osd-deployment.yaml>

      <osd-deployment.yaml>

      Is the OSD deployment yaml file

    • For MGR deployments:

      # oc replace --force -f <mgr-deployment.yaml>

      <mgr-deployment.yaml>

      Is the MGR deployment yaml file

      Important

      Ensure that the MON, MGR and OSD pods are up and running.

18. Scale up the rook-ceph-operator and ocs-operator deployments.

    # oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

    # oc -n openshift-storage scale deployment ocs-operator --replicas=1

1. Scale down the rook-ceph-operator and ocs operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   # oc scale deployment ocs-operator --replicas=0 -n openshift-storage

2. Patch the MDS deployments to remove the livenessProbe parameter and run it with the command parameter as sleep.

   # for i in $(oc get deployment -l app=rook-ceph-mds -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mds", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

3. Recover the CephFS.

   # ceph fs reset ocs-storagecluster-cephfilesystem --yes-i-really-mean-it

   If the reset command fails, force create the default filesystem with the data and metadata pools, and then reset it.

   Note

   The reset command might fail if the cephfilesystem is missing.

   # ceph fs new ocs-storagecluster-cephfilesystem ocs-storagecluster-cephfilesystem-metadata ocs-storagecluster-cephfilesystem-data0 --force

   # ceph fs reset ocs-storagecluster-cephfilesystem --yes-i-really-mean-it

4. Replace the MDS deployments.

   # oc replace --force -f oc_get_deployment.rook-ceph-mds-ocs-storagecluster-cephfilesystem-a.yaml

   # oc replace --force -f oc_get_deployment.rook-ceph-mds-ocs-storagecluster-cephfilesystem-b.yaml

5. Scale up the rook-ceph-operator and ocs-operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=1 -n openshift-storage

   # oc scale deployment ocs-operator --replicas=1 -n openshift-storage

6. Check the CephFS status.

   # ceph fs status

   The status should be active.

1. Check the MCG status.

   noobaa status -n openshift-storage

2. Restart all the pods related to the MCG.

   # oc delete pods <noobaa-operator> -n openshift-storage

   # oc delete pods <noobaa-core> -n openshift-storage

   # oc delete pods <noobaa-endpoint> -n openshift-storage

   # oc delete pods <noobaa-db> -n openshift-storage

   <noobaa-operator>

   Is the name of the MCG operator

   <noobaa-core>

   Is the name of the MCG core pod

   <noobaa-endpoint>

   Is the name of the MCG endpoint

   <noobaa-db>

   Is the name of the MCG db pod

3. If the RADOS Object Gateway (RGW) is configured, restart the pod.

   # oc delete pods <rgw-pod> -n openshift-storage

   <rgw-pod>

   Is the name of the RGW pod