Ce contenu n'est pas disponible dans la langue sélectionnée.

Chapter 12. Restoring the monitor pods in OpenShift Data Foundation

Restore the monitor pods if all three of them go down, and when OpenShift Data Foundation is not able to recover the monitor pods automatically.

Note

This is a disaster recovery procedure and must be performed under the guidance of the Red Hat support team. Contact Red Hat support team on, Red Hat support.

Procedure

Scale down the rook-ceph-operator and ocs operator deployments.

oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

# oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

Copy to Clipboard

Toggle word wrap

oc scale deployment ocs-operator --replicas=0 -n openshift-storage

# oc scale deployment ocs-operator --replicas=0 -n openshift-storage

Copy to Clipboard

Toggle word wrap

Create a backup of all deployments in openshift-storage namespace.

mkdir backup

# mkdir backup

Copy to Clipboard

Toggle word wrap

cd backup

# cd backup

Copy to Clipboard

Toggle word wrap

oc project openshift-storage

# oc project openshift-storage

Copy to Clipboard

Toggle word wrap

for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

# for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

Copy to Clipboard

Toggle word wrap

Patch the Object Storage Device (OSD) deployments to remove the livenessProbe parameter, and run it with the command parameter as sleep.

for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

# for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

Copy to Clipboard

Toggle word wrap

Copy tar to the OSDs.

for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

Copy to Clipboard

Toggle word wrap

Note

While copying the tar binary to the OSD, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues.

Retrieve the monstore cluster map from all the OSDs.

Create the recover_mon.sh script.

#!/bin/bash
ms=/tmp/monstore

rm -rf $ms
mkdir $ms

for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do

  echo "Starting with pod: $osd_pod"

  podname=$(echo $osd_pod|sed 's/pod\///g')
  oc exec $osd_pod -- rm -rf $ms
  oc exec $osd_pod -- mkdir $ms
  oc cp $ms $podname:$ms

  rm -rf $ms
  mkdir $ms

  echo "pod in loop: $osd_pod ; done deleting local dirs"

  oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
  echo "Done with COT on pod: $osd_pod"

  oc cp $podname:$ms $ms

  echo "Finished pulling COT data from pod: $osd_pod"
done

#!/bin/bash
ms=/tmp/monstore

rm -rf $ms
mkdir $ms

for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do

  echo "Starting with pod: $osd_pod"

  podname=$(echo $osd_pod|sed 's/pod\///g')
  oc exec $osd_pod -- rm -rf $ms
  oc exec $osd_pod -- mkdir $ms
  oc cp $ms $podname:$ms

  rm -rf $ms
  mkdir $ms

  echo "pod in loop: $osd_pod ; done deleting local dirs"

  oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
  echo "Done with COT on pod: $osd_pod"

  oc cp $podname:$ms $ms

  echo "Finished pulling COT data from pod: $osd_pod"
done

Copy to Clipboard

Toggle word wrap

Run the recover_mon.sh script.
```
chmod +x recover_mon.sh
```
```
# chmod +x recover_mon.sh
```
Copy to Clipboard Toggle word wrap
```
./recover_mon.sh
```
```
# ./recover_mon.sh
```
Copy to Clipboard Toggle word wrap

Patch the MON deployments, and run it with the command parameter as sleep.

Edit the MON deployments.

for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

# for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

Copy to Clipboard

Toggle word wrap

Patch the MON deployments to increase the initialDelaySeconds.

for i in a b c ; do oc get deployment rook-ceph-mon-${i} -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 10000/g" | oc replace -f - ; done

# for i in a b c ; do oc get deployment rook-ceph-mon-${i} -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 10000/g" | oc replace -f - ; done

Copy to Clipboard

Toggle word wrap

Copy tar to the MON pods.

for i in `oc get pods -l app=rook-ceph-mon -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

# for i in `oc get pods -l app=rook-ceph-mon -o name | sed -e "s/pod\///g"` ; do cat /usr/bin/tar | oc exec -i ${i} -- bash -c 'cat - >/usr/bin/tar' ; oc exec -i ${i} -- bash -c 'chmod +x /usr/bin/tar' ;done

Copy to Clipboard

Toggle word wrap

Note

While copying the tar binary to the MON, it is important to ensure that the tar binary matches the container image OS of the pod. Copying the binary from a different OS such as, macOS, Ubuntu, and so on might lead to compatibility issues.

Copy the previously retrieved monstore to the mon-a pod.

oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

# oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

Copy to Clipboard

Toggle word wrap

Navigate into the MON pod and change the ownership of the retrieved monstore.
```
oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)
```
```
# oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)
```
Copy to Clipboard Toggle word wrap
```
chown -R ceph:ceph /tmp/monstore
```
```
# chown -R ceph:ceph /tmp/monstore
```
Copy to Clipboard Toggle word wrap

Copy the keyring template file before rebuilding the mon db.

oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

# oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

Copy to Clipboard

Toggle word wrap

cp /etc/ceph/keyring-store/keyring /tmp/keyring

# cp /etc/ceph/keyring-store/keyring /tmp/keyring

Copy to Clipboard

Toggle word wrap

cat /tmp/keyring
  [mon.]
    key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
    caps mon = "allow *"
  [client.admin]
    key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
    auid = 0
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *”

# cat /tmp/keyring
  [mon.]
    key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
    caps mon = "allow *"
  [client.admin]
    key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
    auid = 0
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *”

Copy to Clipboard

Toggle word wrap

Populate the keyring of all other Ceph daemons (OSD, MGR, MDS and RGW) from their respective secrets.

oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d

# oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d

[mds.ocs-storagecluster-cephfilesystem-a]
key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
caps mon = "allow profile mds"
caps osd = "allow *"
caps mds = "allow"

Copy to Clipboard

Toggle word wrap

When getting the daemons keyring, use the following command:

for i in `oc get secret | grep keyring| awk '{print $1}'` ; do oc get secret ${i} -ojson  | jq .data.keyring | xargs echo | base64 -d ; done

# for i in `oc get secret | grep keyring| awk '{print $1}'` ; do oc get secret ${i} -ojson  | jq .data.keyring | xargs echo | base64 -d ; done

Copy to Clipboard

Toggle word wrap

Get the OSDs keys with the following script:

for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do oc exec -i ${i} -- bash -c 'cat /var/lib/ceph/osd/ceph-*/keyring ' ;done

# for i in `oc get pods -l app=rook-ceph-osd -o name | sed -e "s/pod\///g"` ; do oc exec -i ${i} -- bash -c 'cat /var/lib/ceph/osd/ceph-*/keyring ' ;done

Copy to Clipboard

Toggle word wrap

Copy the mon keyring locally, then edit it by adding all daemon keys captured in the earlier step and copy it back to one of the MON pods (mon-a):

oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/etc/ceph/keyring-store/..data/keyring /tmp/keyring-mon-a

oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/etc/ceph/keyring-store/..data/keyring /tmp/keyring-mon-a

Copy to Clipboard

Toggle word wrap

vi /tmp/keyring-mon-a

vi /tmp/keyring-mon-a

Copy to Clipboard

Toggle word wrap

As an example, the keyring file should look like the following:

[mon.]
	key = AQCbQLRn0j9mKhAAJKWmMZ483QIpMwzx/yGSLw==
	caps mon = "allow *"
[mds.ocs-storagecluster-cephfilesystem-a]
	key = AQBFQbRnYuB9LxAA8i1fCSAKQQsPuywZ0Jlc5Q==
	caps mon = "allow profile mds"
	caps osd = "allow *"
	caps mds = "allow"
[mds.ocs-storagecluster-cephfilesystem-b]
	key = AQBHQbRnwHAOEBAAv+rBpYP5W8BmC7gLfLyk1w==
	caps mon = "allow profile mds"
	caps osd = "allow *"
	caps mds = "allow"
[osd.0]
    key = AQAvQbRnjF0eEhAA3H0l9zvKGZZM9Up6fJajhQ==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.1]
    key = AQA0QbRnq4cSGxAA7JpuK1+sq8gALNmMYFUMzw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.2]
    key = AQA3QbRn6JvcOBAAFKruZQhlQJKUOi9oxcN6fw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[client.admin]
	key = AQCbQLRnSzOuLBAAK1cSgr2eIyrZV8mV28UfvQ==
	caps mds = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
	caps mgr = "allow *"
[client.rgw.ocs.storagecluster.cephobjectstore.a]
	key = AQBTQbRny7NJLRAAPeTvK9kVg71/glbYLANGyw==
	caps mon = "allow rw"
	caps osd = "allow rwx"
[mgr.a]
	key = AQD9QLRn8+xzDxAARqWQatoT9ruK76EpDS6iCw==
	caps mon = "allow profile mgr"
	caps mds = "allow *"
	caps osd = "allow *"
[mgr.b]
	key = AQD9QLRnltZOIhAAexshUqdOr3G79HWYXUDGFg==
	caps mon = "allow profile mgr"
	caps mds = "allow *"
	caps osd = "allow *"
[client.crash]
	key = AQD7QLRn6DDzCBAAEzhXRzGQUBUNTzC3nHntFQ==
	caps mon = "allow profile crash"
	caps mgr = "allow rw"
[client.ceph-exporter]
	key = AQD7QLRntHzkGxAApQTkMVzcTiZn7jZbwK99SQ==
	caps mon = "allow profile ceph-exporter"
	caps mgr = "allow r"
	caps osd = "allow r"
	caps mds = "allow r"

[mon.]
	key = AQCbQLRn0j9mKhAAJKWmMZ483QIpMwzx/yGSLw==
	caps mon = "allow *"
[mds.ocs-storagecluster-cephfilesystem-a]
	key = AQBFQbRnYuB9LxAA8i1fCSAKQQsPuywZ0Jlc5Q==
	caps mon = "allow profile mds"
	caps osd = "allow *"
	caps mds = "allow"
[mds.ocs-storagecluster-cephfilesystem-b]
	key = AQBHQbRnwHAOEBAAv+rBpYP5W8BmC7gLfLyk1w==
	caps mon = "allow profile mds"
	caps osd = "allow *"
	caps mds = "allow"
[osd.0]
    key = AQAvQbRnjF0eEhAA3H0l9zvKGZZM9Up6fJajhQ==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.1]
    key = AQA0QbRnq4cSGxAA7JpuK1+sq8gALNmMYFUMzw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.2]
    key = AQA3QbRn6JvcOBAAFKruZQhlQJKUOi9oxcN6fw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[client.admin]
	key = AQCbQLRnSzOuLBAAK1cSgr2eIyrZV8mV28UfvQ==
	caps mds = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
	caps mgr = "allow *"
[client.rgw.ocs.storagecluster.cephobjectstore.a]
	key = AQBTQbRny7NJLRAAPeTvK9kVg71/glbYLANGyw==
	caps mon = "allow rw"
	caps osd = "allow rwx"
[mgr.a]
	key = AQD9QLRn8+xzDxAARqWQatoT9ruK76EpDS6iCw==
	caps mon = "allow profile mgr"
	caps mds = "allow *"
	caps osd = "allow *"
[mgr.b]
	key = AQD9QLRnltZOIhAAexshUqdOr3G79HWYXUDGFg==
	caps mon = "allow profile mgr"
	caps mds = "allow *"
	caps osd = "allow *"
[client.crash]
	key = AQD7QLRn6DDzCBAAEzhXRzGQUBUNTzC3nHntFQ==
	caps mon = "allow profile crash"
	caps mgr = "allow rw"
[client.ceph-exporter]
	key = AQD7QLRntHzkGxAApQTkMVzcTiZn7jZbwK99SQ==
	caps mon = "allow profile ceph-exporter"
	caps mgr = "allow r"
	caps osd = "allow r"
	caps mds = "allow r"

Copy to Clipboard

Toggle word wrap

Note

If the caps entries are not present in the OSDs keys output, make sure to add caps to all the OSDs output as mentioned in the previous keyring file example.

oc cp /tmp/keyring-mon-a $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/tmp/keyring

oc cp /tmp/keyring-mon-a $(oc get po -l app=rook-ceph-mon,mon=a -oname|sed -e "s/pod\///g"):/tmp/keyring

Copy to Clipboard

Toggle word wrap

Navigate into the mon-a pod, and verify that the monstore has a monmap.

Navigate into the mon-a pod.

oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

# oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

Copy to Clipboard

Toggle word wrap

Verify that the monstore has a monmap.

ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

# ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

Copy to Clipboard

Toggle word wrap

monmaptool /tmp/monmap --print

# monmaptool /tmp/monmap --print

Copy to Clipboard

Toggle word wrap

Optional: If the monmap is missing then create a new monmap.
```
monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>
```
```
# monmaptool --create --add <mon-a-id> <mon-a-ip> --add <mon-b-id> <mon-b-ip> --add <mon-c-id> <mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>
```
Copy to Clipboard Toggle word wrap
<mon-a-id>
Is the ID of the mon-a pod.
<mon-a-ip>
Is the IP address of the mon-a pod.
<mon-b-id>
Is the ID of the mon-b pod.
<mon-b-ip>
Is the IP address of the mon-b pod.
<mon-c-id>
Is the ID of the mon-c pod.
<mon-c-ip>
Is the IP address of the mon-c pod.
<fsid>
Is the file system ID.
Verify the monmap.
```
monmaptool /root/monmap --print
```
```
# monmaptool /root/monmap --print
```
Copy to Clipboard Toggle word wrap

Import the monmap.

Important

Use the previously created keyring file.

ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

Copy to Clipboard

Toggle word wrap

chown -R ceph:ceph /tmp/monstore

# chown -R ceph:ceph /tmp/monstore

Copy to Clipboard

Toggle word wrap

Create a backup of the old store.db file.

mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

# mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

Copy to Clipboard

Toggle word wrap

mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

# mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

Copy to Clipboard

Toggle word wrap

mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

# mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

Copy to Clipboard

Toggle word wrap

Copy the rebuild store.db file to the monstore directory.

mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

# mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

Copy to Clipboard

Toggle word wrap

chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

# chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

Copy to Clipboard

Toggle word wrap

After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods.

oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

# oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

Copy to Clipboard

Toggle word wrap

oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

# oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

Copy to Clipboard

Toggle word wrap

<id>: Is the ID of the MON pod

Navigate into the rest of the MON pods and change the ownership of the copied monstore.

oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

# oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

Copy to Clipboard

Toggle word wrap

chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

# chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

Copy to Clipboard

Toggle word wrap

<id>: Is the ID of the MON pod

Revert the patched changes.
- For MON deployments:
  # oc replace --force -f <mon-deployment.yaml>
  Copy to Clipboard Toggle word wrap
  <mon-deployment.yaml>
  Is the MON deployment yaml file
- For OSD deployments:
  # oc replace --force -f <osd-deployment.yaml>
  Copy to Clipboard Toggle word wrap
  <osd-deployment.yaml>
  Is the OSD deployment yaml file
- For MGR deployments:
  # oc replace --force -f <mgr-deployment.yaml>
  Copy to Clipboard Toggle word wrap
  <mgr-deployment.yaml>
  Is the MGR deployment yaml file
  Important
  Ensure that the MON, MGR and OSD pods are up and running.

Scale up the rook-ceph-operator and ocs-operator deployments.

oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

# oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

Copy to Clipboard

Toggle word wrap

oc -n openshift-storage scale deployment ocs-operator --replicas=1

# oc -n openshift-storage scale deployment ocs-operator --replicas=1

Copy to Clipboard

Toggle word wrap

Verification steps

Check the Ceph status to confirm that CephFS is running using the odf-cli tool, which you can download from the which you need to download from the customer portal.

odf ceph -s

# odf ceph -s

Copy to Clipboard

Toggle word wrap

Example output:

cluster:
   id:     f111402f-84d1-4e06-9fdb-c27607676e55
   health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            3 daemons have recently crashed

   services:
     mon: 3 daemons, quorum b,c,a (age 15m)
     mgr: a(active, since 14m)
     mds: ocs-storagecluster-cephfilesystem:0
     osd: 3 osds: 3 up (since 15m), 3 in (since 2h)

   data:
     pools:   3 pools, 96 pgs
     objects: 500 objects, 1.1 GiB
     usage:   5.5 GiB used, 295 GiB / 300 GiB avail
     pgs:     96 active+clean

cluster:
   id:     f111402f-84d1-4e06-9fdb-c27607676e55
   health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            3 daemons have recently crashed

   services:
     mon: 3 daemons, quorum b,c,a (age 15m)
     mgr: a(active, since 14m)
     mds: ocs-storagecluster-cephfilesystem:0
     osd: 3 osds: 3 up (since 15m), 3 in (since 2h)

   data:
     pools:   3 pools, 96 pgs
     objects: 500 objects, 1.1 GiB
     usage:   5.5 GiB used, 295 GiB / 300 GiB avail
     pgs:     96 active+clean

Copy to Clipboard

Toggle word wrap

Check the Multicloud Object Gateway (MCG) status. It should be active, and the backingstore and bucketclass should be in Ready state.
```
noobaa status -n openshift-storage
```
```
noobaa status -n openshift-storage
```
Copy to Clipboard Toggle word wrap
Important
If the MCG is not in the active state, and the backingstore and bucketclass not in the Ready state, you need to restart all the MCG related pods. For more information, see Section 12.1, “Restoring the Multicloud Object Gateway”.

12.1. Restoring the Multicloud Object Gateway
Copier lien

If the Multicloud Object Gateway (MCG) is not in the active state, and the backingstore and bucketclass is not in the Ready state, you need to restart all the MCG related pods, and check the MCG status to confirm that the MCG is back up and running.

Procedure

Restart all the pods related to the MCG.
```
oc delete pods <noobaa-operator> -n openshift-storage
```
```
# oc delete pods <noobaa-operator> -n openshift-storage
```
Copy to Clipboard Toggle word wrap
```
oc delete pods <noobaa-core> -n openshift-storage
```
```
# oc delete pods <noobaa-core> -n openshift-storage
```
Copy to Clipboard Toggle word wrap
```
oc delete pods <noobaa-endpoint> -n openshift-storage
```
```
# oc delete pods <noobaa-endpoint> -n openshift-storage
```
Copy to Clipboard Toggle word wrap
```
oc delete pods <noobaa-db> -n openshift-storage
```
```
# oc delete pods <noobaa-db> -n openshift-storage
```
Copy to Clipboard Toggle word wrap
<noobaa-operator>
Is the name of the MCG operator
<noobaa-core>
Is the name of the MCG core pod
<noobaa-endpoint>
Is the name of the MCG endpoint
<noobaa-db>
Is the name of the MCG db pod
If the RADOS Object Gateway (RGW) is configured, restart the pod.
```
oc delete pods <rgw-pod> -n openshift-storage
```
```
# oc delete pods <rgw-pod> -n openshift-storage
```
Copy to Clipboard Toggle word wrap
<rgw-pod>
Is the name of the RGW pod

Note

In OpenShift Container Platform, after the recovery, RBD PVC fails to get mounted on the application pods. Hence, you need to restart the node that is hosting the application pods. To get the node name that is hosting the application pod, run the following command:

oc get pods <application-pod> -n <namespace> -o yaml | grep nodeName

# oc get pods <application-pod> -n <namespace> -o yaml | grep nodeName
  nodeName: node_name

Copy to Clipboard

Toggle word wrap

Ce contenu n'est pas disponible dans la langue sélectionnée.

Chapter 12. Restoring the monitor pods in OpenShift Data Foundation

12.1. Restoring the Multicloud Object Gateway
Copier lien

Apprendre

Essayez, achetez et vendez

Communautés

À propos de la documentation Red Hat

Rendre l’open source plus inclusif

À propos de Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Ce contenu n'est pas disponible dans la langue sélectionnée.

Chapter 12. Restoring the monitor pods in OpenShift Data Foundation

12.1. Restoring the Multicloud Object GatewayCopier lienLien copié sur presse-papiers!

Apprendre

Essayez, achetez et vendez

Communautés

À propos de la documentation Red Hat

Rendre l’open source plus inclusif

À propos de Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

12.1. Restoring the Multicloud Object Gateway
Copier lien