Chapter 5. Troubleshooting Ceph OSDs

Procedure

1. To do so, set the noout flag before stopping the OSD:

   [root@mon ~]# ceph osd set noout

2. When you finish troubleshooting or maintenance, unset the noout flag to start rebalancing:

   [root@mon ~]# ceph osd unset noout

Procedure

1. Mount the partition:

   [root@ceph-mon]# mount -o noatime PARTITION /var/lib/ceph/osd/CLUSTER_NAME-OSD_NUMBER

   Replace PARTITION with the path to the partition on the OSD drive dedicated to OSD data. Specify the cluster name and the OSD number.

   Example

   [root@ceph-mon]# mount -o noatime /dev/sdd1 /var/lib/ceph/osd/ceph-0

2. Try to start the failed ceph-osd daemon:

   [root@ceph-mon]# systemctl start ceph-osd@OSD_NUMBER

   Replace the OSD_NUMBER with the ID of the OSD.

   Example

   [root@ceph-mon]# systemctl start ceph-osd@0

1. Remove the OSD from the Ceph cluster. For details, see the Removing an OSD from the Ceph Cluster procedure.
2. Replace the drive. For details, see Replacing the physical drive] section.
3. Add the OSD to the cluster. For details, see Adding an OSD to the Ceph Cluster] procedure.

Procedure: Removing an OSD from the Ceph Cluster

1. Mark the OSD as out:

   [root@mon ~]# ceph osd out osd.OSD_NUMBER

   Replace OSD_NUMBER with the ID of the OSD that is marked as down, for example:

   [root@mon ~]# ceph osd out osd.0
   marked out osd.0.

   Note

   If the OSD is down, Ceph marks it as out automatically after 600 seconds when it does not receive any heartbeat packet from the OSD. When this happens, other OSDs with copies of the failed OSD data begin backfilling to ensure that the required number of copies exists within the cluster. While the cluster is backfilling, the cluster will be in a degraded state.

2. Ensure that the failed OSD is backfilling. The output will include information similar to the following one:

   [root@mon ~]# ceph -w | grep backfill
   2017-06-02 04:48:03.403872 mon.0 [INF] pgmap v10293282: 431 pgs: 1 active+undersized+degraded+remapped+backfilling, 28 active+undersized+degraded, 49 active+undersized+degraded+remapped+wait_backfill, 59 stale+active+clean, 294 active+clean; 72347 MB data, 101302 MB used, 1624 GB / 1722 GB avail; 227 kB/s rd, 1358 B/s wr, 12 op/s; 10626/35917 objects degraded (29.585%); 6757/35917 objects misplaced (18.813%); 63500 kB/s, 15 objects/s recovering
   2017-06-02 04:48:04.414397 mon.0 [INF] pgmap v10293283: 431 pgs: 2 active+undersized+degraded+remapped+backfilling, 75 active+undersized+degraded+remapped+wait_backfill, 59 stale+active+clean, 295 active+clean; 72347 MB data, 101398 MB used, 1623 GB / 1722 GB avail; 969 kB/s rd, 6778 B/s wr, 32 op/s; 10626/35917 objects degraded (29.585%); 10580/35917 objects misplaced (29.457%); 125 MB/s, 31 objects/s recovering
   2017-06-02 04:48:00.380063 osd.1 [INF] 0.6f starting backfill to osd.0 from (0'0,0'0] MAX to 2521'166639
   2017-06-02 04:48:00.380139 osd.1 [INF] 0.48 starting backfill to osd.0 from (0'0,0'0] MAX to 2513'43079
   2017-06-02 04:48:00.380260 osd.1 [INF] 0.d starting backfill to osd.0 from (0'0,0'0] MAX to 2513'136847
   2017-06-02 04:48:00.380849 osd.1 [INF] 0.71 starting backfill to osd.0 from (0'0,0'0] MAX to 2331'28496
   2017-06-02 04:48:00.381027 osd.1 [INF] 0.51 starting backfill to osd.0 from (0'0,0'0] MAX to 2513'87544

3. Remove the OSD from the CRUSH map:

   [root@mon ~]# ceph osd crush remove osd.OSD_NUMBER

   Replace OSD_NUMBER with the ID of the OSD that is marked as down, for example:

   [root@mon ~]# ceph osd crush remove osd.0
   removed item id 0 name 'osd.0' from crush map

4. Remove authentication keys related to the OSD:

   [root@mon ~]# ceph auth del osd.OSD_NUMBER

   Replace OSD_NUMBER with the ID of the OSD that is marked as down, for example:

   [root@mon ~]# ceph auth del osd.0
   updated

5. Remove the OSD from the Ceph Storage Cluster:

   [root@mon ~]# ceph osd rm osd.OSD_NUMBER

   Replace OSD_NUMBER with the ID of the OSD that is marked as down, for example:

   [root@mon ~]# ceph osd rm osd.0
   removed osd.0

   If you have removed the OSD successfully, it is not present in the output of the following command:

   [root@mon ~]# ceph osd tree

6. For bare-metal deployments, unmount the failed drive:

   [root@mon ~]# umount /var/lib/ceph/osd/CLUSTER_NAME-OSD_NUMBER

   Specify the name of the cluster and the ID of the OSD, for example:

   [root@mon ~]# umount /var/lib/ceph/osd/ceph-0/

   If you have unmounted the drive successfully, it is not present in the output of the following command:

   [root@mon ~]# df -h

1. If the drive is hot-swappable, replace the failed drive with a new one.
2. If the drive is not hot-swappable and the node contains multiple OSDs, you might have to shut down the whole node and replace the physical drive. Consider preventing the cluster from backfilling. See the Stopping and Starting Rebalancing chapter in the Red Hat Ceph Storage Troubleshooting Guide for details.
3. When the drive appears under the /dev/ directory, make a note of the drive path.
4. If you want to add the OSD manually, find the OSD drive and format the disk.

Procedure: Adding an OSD to the Ceph Cluster

1. Add the OSD again.

   1. If you used Ansible to deploy the cluster, run the ceph-ansible playbook again from the Ceph administration server:

      • Bare-metal deployments:

        Syntax

        ansible-playbook site.yml -i hosts --limit NEW_OSD_NODE_NAME

        Example

        [user@admin ceph-ansible]$ ansible-playbook site.yml -i hosts --limit node03

      • Container deployments:

        Syntax

        ansible-playbook site-container.yml -i hosts --limit NEW_OSD_NODE_NAME

        Example

        [user@admin ceph-ansible]$ ansible-playbook site-container.yml -i hosts --limit node03

   2. If you added the OSD manually, see the Adding a Ceph OSD with the Command-line Interface section in the Red Hat Ceph Storage 4 Operations Guide.

2. Ensure that the CRUSH hierarchy is accurate:

   [root@mon ~]# ceph osd tree

3. If you are not satisfied with the location of the OSD in the CRUSH hierarchy, move the OSD to a desired location:

   [root@mon ~]# ceph osd crush move BUCKET_TO_MOVE BUCKET_TYPE=PARENT_BUCKET

   For example, to move the bucket located at sdd:row1 to the root bucket:

   [root@mon ~]# ceph osd crush move ssd:row1 root=ssd:root

Procedure

1. Determine the current value of full_ratio, by default it is set to 0.95:

   [root@mon ~]# ceph osd dump | grep -i full
   full_ratio 0.95

2. Temporarily increase the value of set-full-ratio to 0.97:

   [root@mon ~]# ceph osd set-full-ratio 0.97

   Important

   Red Hat strongly recommends to not set the set-full-ratio to a value higher than 0.97. Setting this parameter to a higher value makes the recovery process harder. As a consequence, you might not be able to recover full OSDs at all.

3. Verify that you successfully set the parameter to 0.97:

   [root@mon ~]# ceph osd dump | grep -i full
   full_ratio 0.97

4. Monitor the cluster state:

   [root@mon ~]# ceph -w

   As soon as the cluster changes its state from full to nearfull, delete any unnecessary data.

5. Set the value of full_ratio back to 0.95:

   [root@mon ~]# ceph osd set-full-ratio 0.95

6. Verify that you successfully set the parameter to 0.95:

   [root@mon ~]# ceph osd dump | grep -i full
   full_ratio 0.95

Procedure

1. Create the partition on the block.db device. This sgdisk command uses the next available partition number automatically:

   Syntax

   sgdisk --new=0:0:_JOURNAL_SIZE_ -- NEW_DEVICE_PATH

   Example

   [root@admin ~]# sgdisk --new=0:0:+2G -- /dev/vdb

2. Create the host_vars directory:

   [root@admin ~]# mkdir /usr/share/ceph-ansible/host_vars

3. Navigate to the host_vars directory:

   [root@admin ~]# cd /usr/share/ceph-ansible/host_vars

4. Create the hosts file on all the hosts of the storage cluster:

   Syntax

   touch NEW_OSD_HOST_NAME

   Example

   [root@admin host_vars]# touch osd5

5. In the hosts file, define the data device:

   Syntax

   lvm_volumes:
   - data: DATA_DEVICE_PATH
     journal: NEW_DEVICE_PARTITION_PATH
   - data: RUNNING_OSD_DATA_DEVICE_PATH
       journal: PARTITION_PATH
   - data: RUNNING_OSD_DATA_DEVICE_PATH
       journal: PARTITION_PATH

   Example

   lvm_volumes:
   - data: /dev/vdd
     journal: /dev/vdb2
   - data: /dev/sdb
     journal: /dev/sdc1
   - data: /dev/sdb
     journal: /dev/sdc2

6. Switch to the Ansible user and verify that Ansible can reach all the Ceph nodes:

   [admin@admin ~]$ ansible all -m ping

7. Change directory to the Ansible configuration directory:

   [admin@admin ~]$ cd /usr/share/ceph-ansible

8. Run the following Ansible playbook with --limit option:

   • Bare-metal deployments:

     [admin@admin ceph-ansible]$ ansible-playbook site.yml --limit osds -i hosts

   • Container deployments:

     [admin@admin ceph-ansible]$ ansible-playbook site-container.yml --limit osds -i hosts