Chapter 12. Restoring the monitor pods in OpenShift Data Foundation

Procedure

1. Scale down the rook-ceph-operator and ocs operator deployments.

   # oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   # oc scale deployment ocs-operator --replicas=0 -n openshift-storage

2. Create a backup of all deployments in openshift-storage namespace.

   # mkdir backup

   # cd backup

   # oc project openshift-storage

   # for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

3. Patch the Object Storage Device (OSD) deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep.

   # for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

4. Copy tar to the OSDs.

   for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

   Note

   While copying the tar binary to the OSD, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues.

5. Retrieve the monstore cluster map from all the OSDs.

   1. Create the recover_mon.sh script.

      #!/bin/bash
      ms=/tmp/monstore
      
      rm -rf $ms
      mkdir $ms
      
      for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do
      
        echo "Starting with pod: $osd_pod"
      
        podname=$(echo $osd_pod|sed 's/pod\///g')
        oc exec $osd_pod -- rm -rf $ms
        oc exec $osd_pod -- mkdir $ms
        oc cp $ms $podname:$ms
      
        rm -rf $ms
        mkdir $ms
      
        echo "pod in loop: $osd_pod ; done deleting local dirs"
      
        oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
        echo "Done with COT on pod: $osd_pod"
      
        oc cp $podname:$ms $ms
      
        echo "Finished pulling COT data from pod: $osd_pod"
      done

   2. Run the recover_mon.sh script.

      # chmod +x recover_mon.sh

      # ./recover_mon.sh

6. Patch the MON deployments, and run it with the command parameter as sleep.

   1. Edit the MON deployments.

      # for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

   2. Patch the MON deployments to increase the initialDelaySeconds.

      # for i in a b c ; do oc get deployment rook-ceph-mon-${i} -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 10000/g" | oc replace -f - ; done

   3. Copy tar to the MON pods.

      # for i in `oc get pods -l app=rook-ceph-mon -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

      Note

      While copying the tar binary to the MON, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues.

7. Copy the previously retrieved monstore to the mon-a pod.

   # oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

8. Navigate into the MON pod and change the ownership of the retrieved monstore.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # chown -R ceph:ceph /tmp/monstore

9. Copy the keyring template file before rebuilding the mon db.

   # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

   # cp /etc/ceph/keyring-store/keyring /tmp/keyring

   # cat /tmp/keyring
     [mon.]
       key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
       caps mon = "allow *"
     [client.admin]
       key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
       auid = 0
       caps mds = "allow *"
       caps mgr = "allow *"
       caps mon = "allow *"
       caps osd = "allow *”

10. Populate the keyring of all other Ceph daemons (OSD, MGR, MDS and RGW) from their respective secrets.

    # oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d
    
    [mds.ocs-storagecluster-cephfilesystem-a]
    key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
    caps mon = "allow profile mds"
    caps osd = "allow *"
    caps mds = "allow"

    When getting the daemons keyring, use the following command:

    # for i in `oc get secret | grep keyring| awk '{print $1}'` ; do oc get secret ${i} -ojson  | jq .data.keyring | xargs echo | base64 -d ; done

    Get the OSDs keys with the following script:

    # for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do oc exec -i ${i} -- bash -c 'cat /var/lib/ceph/osd/ceph-*/keyring ' ;done

    Copy the mon keyring locally, then edit it by adding all daemon keys captured in the earlier step and copy it back to one of the MON pods (mon-a):

    oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/etc/ceph/keyring-store/..data/keyring /tmp/keyring-mon-a

    vi /tmp/keyring-mon-a

    As an example, the keyring file should look like the following:

    [mon.]
    	key = AQCbQLRn0j9mKhAAJKWmMZ483QIpMwzx/yGSLw==
    	caps mon = "allow *"
    [mds.ocs-storagecluster-cephfilesystem-a]
    	key = AQBFQbRnYuB9LxAA8i1fCSAKQQsPuywZ0Jlc5Q==
    	caps mon = "allow profile mds"
    	caps osd = "allow *"
    	caps mds = "allow"
    [mds.ocs-storagecluster-cephfilesystem-b]
    	key = AQBHQbRnwHAOEBAAv+rBpYP5W8BmC7gLfLyk1w==
    	caps mon = "allow profile mds"
    	caps osd = "allow *"
    	caps mds = "allow"
    [osd.0]
        key = AQAvQbRnjF0eEhAA3H0l9zvKGZZM9Up6fJajhQ==
        caps mgr = "allow profile osd"
        caps mon = "allow profile osd"
        caps osd = "allow *"
    [osd.1]
        key = AQA0QbRnq4cSGxAA7JpuK1+sq8gALNmMYFUMzw==
        caps mgr = "allow profile osd"
        caps mon = "allow profile osd"
        caps osd = "allow *"
    [osd.2]
        key = AQA3QbRn6JvcOBAAFKruZQhlQJKUOi9oxcN6fw==
        caps mgr = "allow profile osd"
        caps mon = "allow profile osd"
        caps osd = "allow *"
    [client.admin]
    	key = AQCbQLRnSzOuLBAAK1cSgr2eIyrZV8mV28UfvQ==
    	caps mds = "allow *"
    	caps mon = "allow *"
    	caps osd = "allow *"
    	caps mgr = "allow *"
    [client.rgw.ocs.storagecluster.cephobjectstore.a]
    	key = AQBTQbRny7NJLRAAPeTvK9kVg71/glbYLANGyw==
    	caps mon = "allow rw"
    	caps osd = "allow rwx"
    [mgr.a]
    	key = AQD9QLRn8+xzDxAARqWQatoT9ruK76EpDS6iCw==
    	caps mon = "allow profile mgr"
    	caps mds = "allow *"
    	caps osd = "allow *"
    [mgr.b]
    	key = AQD9QLRnltZOIhAAexshUqdOr3G79HWYXUDGFg==
    	caps mon = "allow profile mgr"
    	caps mds = "allow *"
    	caps osd = "allow *"
    [client.crash]
    	key = AQD7QLRn6DDzCBAAEzhXRzGQUBUNTzC3nHntFQ==
    	caps mon = "allow profile crash"
    	caps mgr = "allow rw"
    [client.ceph-exporter]
    	key = AQD7QLRntHzkGxAApQTkMVzcTiZn7jZbwK99SQ==
    	caps mon = "allow profile ceph-exporter"
    	caps mgr = "allow r"
    	caps osd = "allow r"
    	caps mds = "allow r"

    Note

    If the caps entries are not present in the OSDs keys output, make sure to add caps to all the OSDs output as mentioned in the previous keyring file example.

    oc cp /tmp/keyring-mon-a $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/tmp/keyring

11. Navigate into the mon-a pod, and verify that the monstore has a monmap.

    1. Navigate into the mon-a pod.

       # oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

    2. Verify that the monstore has a monmap.

       # ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

       # monmaptool /tmp/monmap --print

12. Optional: If the monmap is missing then create a new monmap.

    # monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>

    <mon-a-id>

    Is the ID of the mon-a pod.

    <mon-a-ip>

    Is the IP address of the mon-a pod.

    <mon-b-id>

    Is the ID of the mon-b pod.

    <mon-b-ip>

    Is the IP address of the mon-b pod.

    <mon-c-id>

    Is the ID of the mon-c pod.

    <mon-c-ip>

    Is the IP address of the mon-c pod.

    <fsid>

    Is the file system ID.

13. Verify the monmap.

    # monmaptool /root/monmap --print

14. Import the monmap.

    Important

    Use the previously created keyring file.

    # ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

    # chown -R ceph:ceph /tmp/monstore

15. Create a backup of the old store.db file.

    # mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

    # mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

16. Copy the rebuild store.db file to the monstore directory.

    # mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

17. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods.

    # oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

    # oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

    <id>

    Is the ID of the MON pod

18. Navigate into the rest of the MON pods and change the ownership of the copied monstore.

    # oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

    # chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

    <id>

    Is the ID of the MON pod

19. Revert the patched changes.

    • For MON deployments:

      # oc replace --force -f <mon-deployment.yaml>

      <mon-deployment.yaml>

      Is the MON deployment yaml file

    • For OSD deployments:

      # oc replace --force -f <osd-deployment.yaml>

      <osd-deployment.yaml>

      Is the OSD deployment yaml file

    • For MGR deployments:

      # oc replace --force -f <mgr-deployment.yaml>

      <mgr-deployment.yaml>

      Is the MGR deployment yaml file

      Important

      Ensure that the MON, MGR and OSD pods are up and running.

20. Scale up the rook-ceph-operator and ocs-operator deployments.

    # oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

    # oc -n openshift-storage scale deployment ocs-operator --replicas=1

Verification steps

1. Check the Ceph status to confirm that CephFS is running using the odf-cli tool, which you can download from the which you need to download from the customer portal.

   # odf ceph -s

   Example output:

   cluster:
      id:     f111402f-84d1-4e06-9fdb-c27607676e55
      health: HEALTH_ERR
               1 filesystem is offline
               1 filesystem is online with fewer MDS than max_mds
               3 daemons have recently crashed
   
      services:
        mon: 3 daemons, quorum b,c,a (age 15m)
        mgr: a(active, since 14m)
        mds: ocs-storagecluster-cephfilesystem:0
        osd: 3 osds: 3 up (since 15m), 3 in (since 2h)
   
      data:
        pools:   3 pools, 96 pgs
        objects: 500 objects, 1.1 GiB
        usage:   5.5 GiB used, 295 GiB / 300 GiB avail
        pgs:     96 active+clean

2. Check the Multicloud Object Gateway (MCG) status. It should be active, and the backingstore and bucketclass should be in Ready state.

   noobaa status -n openshift-storage

   Important

   If the MCG is not in the active state, and the backingstore and bucketclass not in the Ready state, you need to restart all the MCG related pods. For more information, see Section 12.1, “Restoring the Multicloud Object Gateway”.