Troubleshooting OpenShift Data Foundation

Procedure

1. Specify a blank node selector for the openshift-storage namespace.

   $ oc annotate namespace openshift-storage openshift.io/node-selector=

   Copy to Clipboard Toggle word wrap

2. Delete the original pods generated by the DaemonSets.

   oc delete pod -l app=csi-cephfsplugin -n openshift-storage
   oc delete pod -l app=csi-rbdplugin -n openshift-storage

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to OpenShift Container Platform Web Console.
2. Click Workloads → Secrets

3. To update the ocs-kms-token used for cluster wide encryption:

   1. Set the Project to openshift-storage.
   2. Click ocs-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.
   4. Click Save.

4. To update the ceph-csi-kms-token for a given project or namespace with encrypted persistent volumes:

   1. Select the required Project.
   2. Click ceph-csi-kms-token → Actions → Edit Secret.
   3. Drag and drop or upload your encryption token file in the Value field. The token can either be a file or text that can be copied and pasted.

   4. Click Save.

      Note

      The token can be deleted only after all the encrypted PVCs using the ceph-csi-kms-token have been deleted.

1. Check the storage class associated with OpenShift Data Foundation cluster’s PVCs with the following command:

   $ oc get pvc -n openshift-storage

   Copy to Clipboard Toggle word wrap

2. Check the output. For clusters with Local Storage Operator, the PVCs associated with ocs-deviceset use the storage class localblock. The output looks similar to the following:

   NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   db-noobaa-db-0            Bound    pvc-d96c747b-2ab5-47e2-b07e-1079623748d8   50Gi       RWO            ocs-storagecluster-ceph-rbd   114s
   ocs-deviceset-0-0-lzfrd   Bound    local-pv-7e70c77c                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-1-0-7rggl   Bound    local-pv-b19b3d48                          1769Gi     RWO            localblock                    2m10s
   ocs-deviceset-2-0-znhk8   Bound    local-pv-e9f22cdc                          1769Gi     RWO            localblock                    2m10s

   Copy to Clipboard Toggle word wrap

1. Verify the Ceph health status.

   For more information see: Verifying Ceph cluster is healthy.

2. Based on the provisioning of the OSDs, remove failed or unwanted Ceph OSDs.

   See:

   • Removing failed or unwanted Ceph OSDs in dynamically provisioned Red Hat OpenShift Data Foundation.
   • Removing failed or unwanted Ceph OSDs provisioned using local storage devices.

1. Check if the openshift-storage namespace is stuck in Terminating state upon deletion.

   $ oc get project -n <namespace>

   Copy to Clipboard Toggle word wrap

   Output:

   NAME                DISPLAY NAME   STATUS
   openshift-storage                  Terminating

   Copy to Clipboard Toggle word wrap

2. Check for the NamespaceFinalizersRemaining and NamespaceContentRemaining messages in the STATUS section of the command output and perform the next step for each of the listed resources.

   $ oc get project openshift-storage -o yaml

   Copy to Clipboard Toggle word wrap

   Example output :

   status:
     conditions:
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All resources successfully discovered
       reason: ResourcesDiscovered
       status: "False"
       type: NamespaceDeletionDiscoveryFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All legacy kube types successfully parsed
       reason: ParsedGroupVersions
       status: "False"
       type: NamespaceDeletionGroupVersionParsingFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: All content successfully deleted, may be waiting on finalization
       reason: ContentDeleted
       status: "False"
       type: NamespaceDeletionContentFailure
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some resources are remaining: cephobjectstoreusers.ceph.rook.io has
         1 resource instances'
       reason: SomeResourcesRemain
       status: "True"
       type: NamespaceContentRemaining
     - lastTransitionTime: "2020-07-26T12:32:56Z"
       message: 'Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io
         in 1 resource instances'
       reason: SomeFinalizersRemain
       status: "True"
       type: NamespaceFinalizersRemaining

   Copy to Clipboard Toggle word wrap

3. Delete all the remaining resources listed in the previous step.

   For each of the resources to be deleted, do the following:

   1. Get the object kind of the resource which needs to be removed. See the message in the above output.

      Example :

      message: Some content in the namespace has finalizers remaining: cephobjectstoreuser.ceph.rook.io

      Here cephobjectstoreuser.ceph.rook.io is the object kind.

   2. Get the Object name corresponding to the object kind.

      $ oc get  <Object-kind> -n  <project-name>

      Copy to Clipboard Toggle word wrap

      Example :

      $ oc get cephobjectstoreusers.ceph.rook.io -n openshift-storage

      Copy to Clipboard Toggle word wrap

      Example output:

      NAME                           AGE
      noobaa-ceph-objectstore-user   26h

      Copy to Clipboard Toggle word wrap

   3. Patch the resources.

      $ oc patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

      Copy to Clipboard Toggle word wrap

      Example:

      $ oc patch -n openshift-storage cephobjectstoreusers.ceph.rook.io/noobaa-ceph-objectstore-user \
      --type=merge -p '{"metadata": {"finalizers":null}}'

      Copy to Clipboard Toggle word wrap

      Output:

      cephobjectstoreuser.ceph.rook.io/noobaa-ceph-objectstore-user patched

      Copy to Clipboard Toggle word wrap

4. Verify that the openshift-storage project is deleted.

   $ oc get project openshift-storage

   Copy to Clipboard Toggle word wrap

   Output:

   Error from server (NotFound): namespaces "openshift-storage" not found

   Copy to Clipboard Toggle word wrap

   If the issue persists, reach out to Red Hat Support.

1. Check for CephFS pvc stuck in Pending status.

   # oc get pvc -n <namespace>

   Copy to Clipboard Toggle word wrap

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Pending
                             ocs-external-storagecluster-cephfs  28h
   [...]

   Copy to Clipboard Toggle word wrap

2. Check the describe output to see the events for respective pvc.

   Expected error message is cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

   # oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

   Copy to Clipboard Toggle word wrap

   Example output:

   Name:          ngx-fs-pxknkcix20-pod
   Namespace:     nginx-file
   StorageClass:  ocs-external-storagecluster-cephfs
   Status:        Pending
   Volume:
   Labels:        <none>
   Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:
   Access Modes:
   VolumeMode:    Filesystem
   Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
   Events:
    Type     Reason              Age                   From                                                                                                                      Message
    ----     ------              ----                  ----                                                                                                                      -------
    Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
    (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
    running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
    csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
    cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)

   Copy to Clipboard Toggle word wrap

3. Check the settings for the <cephfs metadata pool name> (here cephfs_metadata ) and <cephfs data pool name> (here cephfs_data). For running the command, you will need jq preinstalled in the Red Hat Ceph Storage client node.

   # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
   {
     "cephfs": {}
   }
   "cephfs_metadata"
   {
      "cephfs": {}
   }

   Copy to Clipboard Toggle word wrap

4. Set the application type for CephFS pool.

   • Run the following commands on the Red Hat Ceph Storage client node :

     # ceph osd pool application set <cephfs metadata pool name> cephfs metadata cephfs

     Copy to Clipboard Toggle word wrap

     # ceph osd pool application set <cephfs data pool name> cephfs data cephfs

     Copy to Clipboard Toggle word wrap

5. Verify if the settings are applied.

    # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
     {
       "cephfs": {
         "data": "cephfs"
      }
     }
     "cephfs_metadata"
     {
       "cephfs": {
         "metadata": "cephfs"
       }
     }

   Copy to Clipboard Toggle word wrap

6. Check the CephFS PVC status again. The PVC should now be in Bound state.

   # oc get pvc -n <namespace>

   Copy to Clipboard Toggle word wrap

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
   1Mi       RWO             ocs-external-storagecluster-cephfs  29h
   [...]

   Copy to Clipboard Toggle word wrap

Procedure

1. Scale down the rook-ceph-operator and ocs operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   Copy to Clipboard Toggle word wrap

   # oc scale deployment ocs-operator --replicas=0 -n openshift-storage

   Copy to Clipboard Toggle word wrap

2. Create a backup of all deployments in openshift-storage namespace.

   # mkdir backup

   Copy to Clipboard Toggle word wrap

   # cd backup

   Copy to Clipboard Toggle word wrap

   # oc project openshift-storage

   Copy to Clipboard Toggle word wrap

   # for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

   Copy to Clipboard Toggle word wrap

3. Patch the OSD deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep.

   # for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

   Copy to Clipboard Toggle word wrap

4. Retrieve the monstore cluster map from all the OSDs.

   1. Create the recover_mon.sh script.

      #!/bin/bash
      ms=/tmp/monstore
      
      rm -rf $ms
      mkdir $ms
      
      for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do
      
        echo "Starting with pod: $osd_pod"
      
        podname=$(echo $osd_pod|sed 's/pod\///g')
        oc exec $osd_pod -- rm -rf $ms
        oc cp $ms $podname:$ms
      
        rm -rf $ms
        mkdir $ms
      
        echo "pod in loop: $osd_pod ; done deleting local dirs"
      
        oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
        echo "Done with COT on pod: $osd_pod"
      
        oc cp $podname:$ms $ms
      
        echo "Finished pulling COT data from pod: $osd_pod"
      done

      Copy to Clipboard Toggle word wrap

   2. Run the recover_mon.sh script.

      # chmod +x recover_mon.sh

      Copy to Clipboard Toggle word wrap

      # ./recover_mon.sh

      Copy to Clipboard Toggle word wrap

5. Patch the MON deployments, and run it with the command parameter as sleep.

   1. Edit the MON deployments.

      # for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

      Copy to Clipboard Toggle word wrap

   2. Patch the MON deployments to increase the initialDelaySeconds.

      # oc get deployment rook-ceph-mon-a -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      Copy to Clipboard Toggle word wrap

      # oc get deployment rook-ceph-mon-b -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      Copy to Clipboard Toggle word wrap

      # oc get deployment rook-ceph-mon-c -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      Copy to Clipboard Toggle word wrap

6. Copy the previously retrieved monstore to the mon-a pod.

   # oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

   Copy to Clipboard Toggle word wrap

7. Navigate into the MON pod and change the ownership of the retrieved monstore.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   Copy to Clipboard Toggle word wrap

   # chown -R ceph:ceph /tmp/monstore

   Copy to Clipboard Toggle word wrap

8. Copy the keyring template file before rebuilding the mon db.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   Copy to Clipboard Toggle word wrap

   # cp /etc/ceph/keyring-store/keyring /tmp/keyring

   Copy to Clipboard Toggle word wrap

   # cat /tmp/keyring
     [mon.]
       key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
       caps mon = "allow *"
     [client.admin]
       key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
       auid = 0
       caps mds = "allow *"
       caps mgr = "allow *"
       caps mon = "allow *"
       caps osd = "allow *”

   Copy to Clipboard Toggle word wrap

9. Identify the keyring of all other Ceph daemons (MGR, MDS, RGW, Crash, CSI and CSI provisioners) from its respective secrets.

   # oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d
   
   [mds.ocs-storagecluster-cephfilesystem-a]
   key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
   caps mon = "allow profile mds"
   caps osd = "allow *"
   caps mds = "allow"

   Copy to Clipboard Toggle word wrap

   Example keyring file, /etc/ceph/ceph.client.admin.keyring:

   [mon.]
   	key = AQDxTF1hNgLTNxAAi51cCojs01b4I5E6v2H8Uw==
   	caps mon = "allow "
   [client.admin]
           key = AQDxTF1hpzguOxAA0sS8nN4udoO35OEbt3bqMQ==
           caps mds = "allow " caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mds.ocs-storagecluster-cephfilesystem-a] key = AQCKTV1horgjARAA8aF/BDh/4+eG4RCNBCl+aw== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [mds.ocs-storagecluster-cephfilesystem-b] key = AQCKTV1hN4gKLBAA5emIVq3ncV7AMEM1c1RmGA== caps mds = "allow" caps mon = "allow profile mds" caps osd = "allow *" [client.rgw.ocs.storagecluster.cephobjectstore.a] key = AQCOkdBixmpiAxAA4X7zjn6SGTI9c1MBflszYA== caps mon = "allow rw" caps osd = "allow rwx" [mgr.a] key = AQBOTV1hGYOEORAA87471+eIZLZtptfkcHvTRg== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [client.crash] key = AQBOTV1htO1aGRAAe2MPYcGdiAT+Oo4CNPSF1g== caps mgr = "allow rw" caps mon = "allow profile crash" [client.csi-cephfs-node] key = AQBOTV1hiAtuBBAAaPPBVgh1AqZJlDeHWdoFLw== caps mds = "allow rw" caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs *=" [client.csi-cephfs-provisioner] key = AQBNTV1hHu6wMBAAzNXZv36aZJuE1iz7S7GfeQ== caps mgr = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs metadata="
   [client.csi-rbd-node]
   	key = AQBNTV1h+LnkIRAAWnpIN9bUAmSHOvJ0EJXHRw==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"
   [client.csi-rbd-provisioner]
   	key = AQBNTV1hMNcsExAAvA3gHB2qaY33LOdWCvHG/A==
   	caps mgr = "allow rw"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd"

   Copy to Clipboard Toggle word wrap
   Important
   • For client.csi related keyring, refer to the previous keyring file output and add the default caps after fetching the key from its respective OpenShift Data Foundation secret.
   • OSD keyring is added automatically post recovery.

10. Navigate into the mon-a pod, and verify that the monstore has monmap.

    1. Navigate into the mon-a pod.

       # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

       Copy to Clipboard Toggle word wrap

    2. Verify that the monstore has monmap.

       # ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

       Copy to Clipboard Toggle word wrap

       # monmaptool /tmp/monmap --print

       Copy to Clipboard Toggle word wrap

11. Optional: If the monmap is missing then create a new monmap.

    # monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>

    Copy to Clipboard Toggle word wrap

    <mon-a-id>

    Is the ID of the mon-a pod.

    <mon-a-ip>

    Is the IP address of the mon-a pod.

    <mon-b-id>

    Is the ID of the mon-b pod.

    <mon-b-ip>

    Is the IP address of the mon-b pod.

    <mon-c-id>

    Is the ID of the mon-c pod.

    <mon-c-ip>

    Is the IP address of the mon-c pod.

    <fsid>

    Is the file system ID.

12. Verify the monmap.

    # monmaptool /root/monmap --print

    Copy to Clipboard Toggle word wrap

13. Import the monmap.

    Important

    Use the previously created keyring file.

    # ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

    Copy to Clipboard Toggle word wrap

    # chown -R ceph:ceph /tmp/monstore

    Copy to Clipboard Toggle word wrap

14. Create a backup of the old store.db file.

    # mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

    Copy to Clipboard Toggle word wrap

    # mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

    Copy to Clipboard Toggle word wrap

    # mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

    Copy to Clipboard Toggle word wrap

15. Copy the rebuild store.db file to the monstore directory.

    # mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

    Copy to Clipboard Toggle word wrap

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

    Copy to Clipboard Toggle word wrap

16. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods.

    # oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

    Copy to Clipboard Toggle word wrap

    # oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

    Copy to Clipboard Toggle word wrap

    <id>

    Is the ID of the MON pod

17. Navigate into the rest of the MON pods and change the ownership of the copied monstore.

    # oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

    Copy to Clipboard Toggle word wrap

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

    Copy to Clipboard Toggle word wrap

    <id>

    Is the ID of the MON pod

18. Revert the patched changes.

    • For MON deployments:

      # oc replace --force -f <mon-deployment.yaml>

      Copy to Clipboard Toggle word wrap

      <mon-deployment.yaml>

      Is the MON deployment yaml file

    • For OSD deployments:

      # oc replace --force -f <osd-deployment.yaml>

      Copy to Clipboard Toggle word wrap

      <osd-deployment.yaml>

      Is the OSD deployment yaml file

    • For MGR deployments:

      # oc replace --force -f <mgr-deployment.yaml>

      Copy to Clipboard Toggle word wrap

      <mgr-deployment.yaml>

      Is the MGR deployment yaml file

      Important

      Ensure that the MON, MGR and OSD pods are up and running.

19. Scale up the rook-ceph-operator and ocs-operator deployments.

    # oc -n openshift-storage scale deployment ocs-operator --replicas=1

    Copy to Clipboard Toggle word wrap

Verification steps

1. Check the Ceph status to confirm that CephFS is running.

   # ceph -s

   Copy to Clipboard Toggle word wrap

   Example output:

   cluster:
      id:     f111402f-84d1-4e06-9fdb-c27607676e55
      health: HEALTH_ERR
               1 filesystem is offline
               1 filesystem is online with fewer MDS than max_mds
               3 daemons have recently crashed
   
      services:
        mon: 3 daemons, quorum b,c,a (age 15m)
        mgr: a(active, since 14m)
        mds: ocs-storagecluster-cephfilesystem:0
        osd: 3 osds: 3 up (since 15m), 3 in (since 2h)
   
      data:
        pools:   3 pools, 96 pgs
        objects: 500 objects, 1.1 GiB
        usage:   5.5 GiB used, 295 GiB / 300 GiB avail
        pgs:     96 active+clean

   Copy to Clipboard Toggle word wrap

2. Check the Multicloud Object Gateway (MCG) status. It should be active, and the backingstore and bucketclass should be in Ready state.

   noobaa status -n openshift-storage

   Copy to Clipboard Toggle word wrap
   Important

   If the MCG is not in the active state, and the backingstore and bucketclass not in the Ready state, you need to restart all the MCG related pods. For more information, see Section 11.1, “Restoring the Multicloud Object Gateway”.

Procedure

1. Stop the rook-ceph-operator so that the mons are not failed over when you are modifying the monmap.

   # oc -n openshift-storage scale deployment rook-ceph-operator --replicas=0

   Copy to Clipboard Toggle word wrap

2. Inject a new monmap.

   Warning

   You must inject the monmap very carefully. If run incorrectly, your cluster could be permanently destroyed. The Ceph monmap keeps track of the mon quorum. The monmap is updated to only contain the healthy mon. In this example, the healthy mon is rook-ceph-mon-b, while the unhealthy mons are rook-ceph-mon-a and rook-ceph-mon-c.

   1. Take a backup of the current rook-ceph-mon-b Deployment:

      # oc -n openshift-storage get deployment rook-ceph-mon-b -o yaml > rook-ceph-mon-b-deployment.yaml

      Copy to Clipboard Toggle word wrap

   2. Open the YAML file and copy the command and arguments from the mon container (see containers list in the following example). This is needed for the monmap changes.

      [...]
        containers:
        - args:
          - --fsid=41a537f2-f282-428e-989f-a9e07be32e47
          - --keyring=/etc/ceph/keyring-store/keyring
          - --log-to-stderr=true
          - --err-to-stderr=true
          - --mon-cluster-log-to-stderr=true
          - '--log-stderr-prefix=debug '
          - --default-log-to-file=false
          - --default-mon-cluster-log-to-file=false
          - --mon-host=$(ROOK_CEPH_MON_HOST)
          - --mon-initial-members=$(ROOK_CEPH_MON_INITIAL_MEMBERS)
          - --id=b
          - --setuser=ceph
          - --setgroup=ceph
          - --foreground
          - --public-addr=10.100.13.242
          - --setuser-match-path=/var/lib/ceph/mon/ceph-b/store.db
          - --public-bind-addr=$(ROOK_POD_IP)
          command:
          - ceph-mon
      [...]

      Copy to Clipboard Toggle word wrap

   3. Cleanup the copied command and args fields to form a pastable command as follows:

      # ceph-mon \
          --fsid=41a537f2-f282-428e-989f-a9e07be32e47 \
          --keyring=/etc/ceph/keyring-store/keyring \
          --log-to-stderr=true \
          --err-to-stderr=true \
          --mon-cluster-log-to-stderr=true \
          --log-stderr-prefix=debug \
          --default-log-to-file=false \
          --default-mon-cluster-log-to-file=false \
          --mon-host=$ROOK_CEPH_MON_HOST \
          --mon-initial-members=$ROOK_CEPH_MON_INITIAL_MEMBERS \
          --id=b \
          --setuser=ceph \
          --setgroup=ceph \
          --foreground \
          --public-addr=10.100.13.242 \
          --setuser-match-path=/var/lib/ceph/mon/ceph-b/store.db \
          --public-bind-addr=$ROOK_POD_IP

      Copy to Clipboard Toggle word wrap
      Note

      Make sure to remove the single quotes around the --log-stderr-prefix flag and the parenthesis around the variables being passed ROOK_CEPH_MON_HOST, ROOK_CEPH_MON_INITIAL_MEMBERS and ROOK_POD_IP).

   4. Patch the rook-ceph-mon-b Deployment to stop the working of this mon without deleting the mon pod.

      # oc -n openshift-storage patch deployment rook-ceph-mon-b  --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]'
      
      # oc -n openshift-storage patch deployment rook-ceph-mon-b -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'

      Copy to Clipboard Toggle word wrap

   5. Perform the following steps on the mon-b pod:

      1. Connect to the pod of a healthy mon and run the following commands:

         # oc -n openshift-storage exec -it <mon-pod> bash

         Copy to Clipboard Toggle word wrap

      2. Set the variable.

         # monmap_path=/tmp/monmap

         Copy to Clipboard Toggle word wrap

      3. Extract the monmap to a file, by pasting the ceph mon command from the good mon deployment and adding the --extract-monmap=${monmap_path} flag.

         # ceph-mon \
                --fsid=41a537f2-f282-428e-989f-a9e07be32e47 \
                --keyring=/etc/ceph/keyring-store/keyring \
                --log-to-stderr=true \
                --err-to-stderr=true \
                --mon-cluster-log-to-stderr=true \
                --log-stderr-prefix=debug \
                --default-log-to-file=false \
                --default-mon-cluster-log-to-file=false \
                --mon-host=$ROOK_CEPH_MON_HOST \
                --mon-initial-members=$ROOK_CEPH_MON_INITIAL_MEMBERS \
                --id=b \
                --setuser=ceph \
                --setgroup=ceph \
                --foreground \
                --public-addr=10.100.13.242 \
                --setuser-match-path=/var/lib/ceph/mon/ceph-b/store.db \
                --public-bind-addr=$ROOK_POD_IP \
                --extract-monmap=${monmap_path}

         Copy to Clipboard Toggle word wrap

      4. Review the contents of the monmap.

         # monmaptool --print /tmp/monmap

         Copy to Clipboard Toggle word wrap

      5. Remove the bad mons from the monmap.

         # monmaptool ${monmap_path} --rm <bad_mon>

         Copy to Clipboard Toggle word wrap

         In this example we remove mon0 and mon2:

         # monmaptool ${monmap_path} --rm a
         # monmaptool ${monmap_path} --rm c

         Copy to Clipboard Toggle word wrap

      6. Inject the modified monmap into the good mon, by pasting the ceph mon command and adding the --inject-monmap=${monmap_path} flag as follows:

         # ceph-mon \
                --fsid=41a537f2-f282-428e-989f-a9e07be32e47 \
                --keyring=/etc/ceph/keyring-store/keyring \
                --log-to-stderr=true \
                --err-to-stderr=true \
                --mon-cluster-log-to-stderr=true \
                --log-stderr-prefix=debug \
                --default-log-to-file=false \
                --default-mon-cluster-log-to-file=false \
                --mon-host=$ROOK_CEPH_MON_HOST \
                --mon-initial-members=$ROOK_CEPH_MON_INITIAL_MEMBERS \
                --id=b \
                --setuser=ceph \
                --setgroup=ceph \
                --foreground \
                --public-addr=10.100.13.242 \
                --setuser-match-path=/var/lib/ceph/mon/ceph-b/store.db \
                --public-bind-addr=$ROOK_POD_IP \
                --inject-monmap=${monmap_path}

         Copy to Clipboard Toggle word wrap
      7. Exit the shell to continue.

3. Edit the Rook configmaps.

   1. Edit the configmap that the operator uses to track the mons.

      # oc -n openshift-storage edit configmap rook-ceph-mon-endpoints

      Copy to Clipboard Toggle word wrap

   2. Verify that in the data element you see three mons such as the following (or more depending on your moncount):

      data: a=10.100.35.200:6789;b=10.100.13.242:6789;c=10.100.35.12:6789

      Copy to Clipboard Toggle word wrap

   3. Delete the bad mons from the list to end up with a single good mon. For example:

      data: b=10.100.13.242:6789

      Copy to Clipboard Toggle word wrap
   4. Save the file and exit.

   5. Now, you need to adapt a Secret which is used for the mons and other components.

      1. Set a value for the variable good_mon_id.

         For example:

         # good_mon_id=b

         Copy to Clipboard Toggle word wrap

      2. You can use the oc patch command to patch the rook-ceph-config secret and update the two key/value pairs mon_host and mon_initial_members.

         # mon_host=$(oc -n openshift-storage get svc rook-ceph-mon-b -o jsonpath='{.spec.clusterIP}')
         
         # oc -n openshift-storage patch secret rook-ceph-config -p '{"stringData": {"mon_host": "[v2:'"${mon_host}"':3300,v1:'"${mon_host}"':6789]", "mon_initial_members": "'"${good_mon_id}"'"}}'

         Copy to Clipboard Toggle word wrap
         Note

         If you are using hostNetwork: true, you need to replace the mon_host var with the node IP the mon is pinned to (nodeSelector). This is because there is no rook-ceph-mon-* service created in that “mode”.

4. Restart the mon.

   You need to restart the good mon pod with the original ceph-mon command to pick up the changes.

   1. Use the oc replace command on the backup of the mon deployment YAML file:

      # oc replace --force -f rook-ceph-mon-b-deployment.yaml

      Copy to Clipboard Toggle word wrap
      Note

      Option --force deletes the deployment and creates a new one.

   2. Verify the status of the cluster.

      The status should show one mon in quorum. If the status looks good, your cluster should be healthy again.

5. Delete the two mon deployments that are no longer expected to be in quorum.

   For example:

   # oc delete deploy <rook-ceph-mon-1>
   # oc delete deploy <rook-ceph-mon-2>

   Copy to Clipboard Toggle word wrap

   In this example the deployments to be deleted are rook-ceph-mon-a and rook-ceph-mon-c.

6. Restart the operator.

   1. Start the rook operator again to resume monitoring the health of the cluster.

      Note

      It is safe to ignore the errors that a number of resources already exist.

      # oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

      Copy to Clipboard Toggle word wrap

      The operator automatically adds more mons to increase the quorum size again depending on the mon count.