Chapter 3. Administering Ceph Clusters That Run in Containers

Procedure

1. To start, stop, or restart a Ceph daemon running in a container, run a systemctl command as root composed in the following format:

   systemctl action ceph-daemon@ID

   Copy to Clipboard Toggle word wrap

   Where:

   • action is the action to perform; start, stop, or restart
   • daemon is the daemon; osd, mon, mds, or rgw

   • ID is either

     • The short host name where the ceph-mon, ceph-mds, or ceph-rgw daemons are running
     • The ID of the ceph-osd daemon if it was deployed the osd_scenario parameter set to lvm
     • The device name that the ceph-osd daemon uses if it was deployed with the osd_scenario parameter set to collocated or non-collocated

   For example, to restart a ceph-osd daemon with the ID osd01:

   # systemctl restart ceph-osd@osd01

   Copy to Clipboard Toggle word wrap

   To start a ceph-mon demon that runs on the ceph-monitor01 host:

   # systemctl start ceph-mon@ceph-monitor01

   Copy to Clipboard Toggle word wrap

   To stop a ceph-rgw daemon that runs on the ceph-rgw01 host:

   # systemctl stop ceph-radosgw@ceph-rgw01

   Copy to Clipboard Toggle word wrap

2. Verify that the action was completed successfully.

   systemctl status ceph-daemon@_ID

   Copy to Clipboard Toggle word wrap

   For example:

   # systemctl status ceph-mon@ceph-monitor01

   Copy to Clipboard Toggle word wrap

Procedure

1. To view the entire Ceph log file, run a journalctl command as root composed in the following format:

   journalctl -u ceph-daemon@ID

   Copy to Clipboard Toggle word wrap

   Where:

   • daemon is the Ceph daemon; osd, mon, or rgw

   • ID is either

     • The short host name where the ceph-mon, ceph-mds, or ceph-rgw daemons are running
     • The ID of the ceph-osd daemon if it was deployed the osd_scenario parameter set to lvm
     • The device name that the ceph-osd daemon uses if it was deployed with the osd_scenario parameter set to collocated or non-collocated

   For example, to view the entire log for the ceph-osd daemon with the ID osd01:

   # journalctl -u ceph-osd@osd01

   Copy to Clipboard Toggle word wrap

2. To show only the recent journal entries, use the -f option.

   journalctl -fu ceph-daemon@ID

   Copy to Clipboard Toggle word wrap

   For example, to view only recent journal entries for the ceph-mon daemon that runs on the ceph-monitor01 host:

   # journalctl -fu ceph-mon@ceph-monitor01

   Copy to Clipboard Toggle word wrap

1. Install the ceph-osd package and create a new OSD instance
2. Prepare and mount the OSD data and journal drives
3. Add the new OSD node to the CRUSH map
4. Update the owner and group permissions
5. Enable and start the ceph-osd daemon

Procedure

1. Enable the Red Hat Ceph Storage 3 OSD software repository.

   Red Hat Enterprise Linux

   [root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd- els-rpms

   Copy to Clipboard Toggle word wrap

2. Create the /etc/ceph/ directory:
3. On the new OSD node, copy the Ceph administration keyring and configuration files from one of the Ceph Monitor nodes:

4. Install the ceph-osd package on the new Ceph OSD node:

   Red Hat Enterprise Linux

   [root@osd ~]# yum install ceph-osd

   Copy to Clipboard Toggle word wrap

5. Decide if you want to collocate a journal or use a dedicated journal for the new OSDs.

   Note

   The --filestore option is required.

   1. For OSDs with a collocated journal:

      Syntax

      [root@osd ~]# docker exec $CONTAINER_ID ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/$DEVICE_NAME

      Copy to Clipboard Toggle word wrap

      Example:

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/sda

      Copy to Clipboard Toggle word wrap

   2. For OSDs with a dedicated journal:

      Syntax

      [root@osd ~]# docker exec $CONTAINER_ID ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/$DEVICE_NAME /dev/$JOURNAL_DEVICE_NAME

      Copy to Clipboard Toggle word wrap

      or

      [root@osd ~]# docker exec $CONTAINER_ID ceph-volume lvm prepare  --filestore --data /dev/$DEVICE_NAME --journal /dev/$JOURNAL_DEVICE_NAME

      Copy to Clipboard Toggle word wrap

      Examples

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/sda /dev/sdb

      Copy to Clipboard Toggle word wrap

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-volume lvm prepare  --filestore --data /dev/vg00/lvol1 --journal /dev/sdb

      Copy to Clipboard Toggle word wrap

6. Set the noup option:

   [root@osd ~]# ceph osd set noup

   Copy to Clipboard Toggle word wrap

7. Activate the new OSD:

   Syntax

   [root@osd ~]# docker exec $CONTAINER_ID ceph-disk activate /dev/$DEVICE_NAME

   Copy to Clipboard Toggle word wrap

   or

   [root@osd ~]# docker exec $CONTAINER_ID ceph-volume lvm activate --filestore $OSD_ID $OSD_FSID

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk activate /dev/sda

   Copy to Clipboard Toggle word wrap

   [root@osd ~]# docker exec ceph-osd-osd1 ceph-volume lvm activate --filestore 0 6cc43680-4f6e-4feb-92ff-9c7ba204120e

   Copy to Clipboard Toggle word wrap

8. Add the OSD to the CRUSH map:

   Syntax

   ceph osd crush add $OSD_ID $WEIGHT [$BUCKET_TYPE=$BUCKET_NAME ...]

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph osd crush add 4 1 host=node4

   Copy to Clipboard Toggle word wrap

   Note

   If you specify more than one bucket, the command places the OSD into the most specific bucket out of those you specified, and it moves the bucket underneath any other buckets you specified.

   Note

   You can also edit the CRUSH map manually. See the Editing a CRUSH map section in the Storage Strategies guide for Red Hat Ceph Storage 3.

   Important

   If you specify only the root bucket, then the OSD attaches directly to the root, but the CRUSH rules expect OSDs to be inside of the host bucket.

9. Unset the noup option:

   [root@osd ~]# ceph osd unset noup

   Copy to Clipboard Toggle word wrap

10. Update the owner and group permissions for the newly created directories:

    Syntax

    chown -R $OWNER:$GROUP $PATH_TO_DIRECTORY

    Copy to Clipboard Toggle word wrap

    Example

    [root@osd ~]# chown -R ceph:ceph /var/lib/ceph/osd
    [root@osd ~]# chown -R ceph:ceph /var/log/ceph
    [root@osd ~]# chown -R ceph:ceph /var/run/ceph
    [root@osd ~]# chown -R ceph:ceph /etc/ceph

    Copy to Clipboard Toggle word wrap

11. If you use clusters with custom names, then add the following line to the appropriate file:

    Red Hat Enterprise Linux

    [root@osd ~]# echo "CLUSTER=$CLUSTER_NAME" >> /etc/sysconfig/ceph

    Copy to Clipboard Toggle word wrap

    Replace $CLUSTER_NAME with the custom cluster name.

12. To ensure that the new OSD is up and ready to receive data, enable and start the OSD service:

    Syntax

    systemctl enable ceph-osd@$OSD_ID
    systemctl start ceph-osd@$OSD_ID

    Copy to Clipboard Toggle word wrap

    Example

    [root@osd ~]# systemctl enable ceph-osd@4
    [root@osd ~]# systemctl start ceph-osd@4

    Copy to Clipboard Toggle word wrap

Procedure

1. Disable and stop the OSD service:

   Syntax

   systemctl disable ceph-osd@$DEVICE_NAME
   systemctl stop ceph-osd@$DEVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl disable ceph-osd@sdb
   [root@osd ~]# systemctl stop ceph-osd@sdb

   Copy to Clipboard Toggle word wrap

   Once the OSD is stopped, it is down.

2. Remove the OSD from the storage cluster:

   Syntax

   ceph osd out $DEVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph osd out sdb

   Copy to Clipboard Toggle word wrap

   Important

   Once the OSD is out, Ceph will start rebalancing and copying data to other OSDs in the storage cluster. Red Hat recommends waiting until the storage cluster becomes active+clean before proceeding to the next step. To observe the data migration, run the following command:

   [root@monitor ~]# ceph -w

   Copy to Clipboard Toggle word wrap

3. Remove the OSD from the CRUSH map so that it no longer receives data.

   Syntax

   ceph osd crush remove $OSD_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph osd crush remove osd.4

   Copy to Clipboard Toggle word wrap

   Note

   You can also decompile the CRUSH map, remove the OSD from the device list, remove the device as an item in the host bucket or remove the host bucket. If it is in the CRUSH map and you intend to remove the host, recompile the map and set it. See the Storage Strategies Guide for details.

4. Remove the OSD authentication key:

   Syntax

   ceph auth del osd.$DEVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph auth del osd.sdb

   Copy to Clipboard Toggle word wrap

5. Remove the OSD:

   Syntax

   ceph osd rm $DEVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph osd rm sdb

   Copy to Clipboard Toggle word wrap

6. Edit the storage cluster’s configuration file, by default /etc/ceph.conf, and remove the OSD entry, if it exists:

   Example

   [osd.4]
   host = $HOST_NAME

   Copy to Clipboard Toggle word wrap

7. Remove the reference to the OSD in the /etc/fstab file, if the OSD was added manually.

8. Copy the updated configuration file to the /etc/ceph/ directory of all other nodes in the storage cluster.

   Syntax

   scp /etc/ceph/$CLUSTER_NAME.conf $USER_NAME@$HOST_NAME:/etc/ceph/

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# scp /etc/ceph/ceph.conf root@node4:/etc/ceph/

   Copy to Clipboard Toggle word wrap

Procedure

1. Destroy the OSD:

   ceph osd destroy $OSD_ID --yes-i-really-mean-it

   Copy to Clipboard Toggle word wrap

   Example

   $ ceph osd destroy 1 --yes-i-really-mean-it

   Copy to Clipboard Toggle word wrap

2. Optionally, if the replacement disk was used previously, then you need to zap the disk:

   docker exec $CONTAINER_ID ceph-volume lvm zap $DEVICE

   Copy to Clipboard Toggle word wrap

   Example

   $ docker exec ceph-osd-osd1 ceph-volume lvm zap /dev/sdb

   Copy to Clipboard Toggle word wrap

3. Create the new OSD with the existing OSD ID:

   docker exec $CONTAINER_ID ceph-volume lvm create --osd-id $OSD_ID --data $DEVICE

   Copy to Clipboard Toggle word wrap

   Example

   $ docker exec ceph-osd-osd1 ceph-volume lvm create --osd-id 1 --data /dev/sdb

   Copy to Clipboard Toggle word wrap

1. As the root user, navigate to the /usr/share/ceph-ansible/ directory.

   [root@admin ~]# cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

2. Copy the purge-docker-cluster.yml playbook from the /usr/share/infrastructure-playbooks/ directory to the current directory:

   [root@admin ceph-ansible]# cp infrastructure-playbooks/purge-docker-cluster.yml .

   Copy to Clipboard Toggle word wrap

3. As the Ansible user, use the purge-docker-cluster.yml playbook to purge the Ceph cluster.

   1. To remove all packages, containers, configuration files, and all the data created by the ceph-ansible playbook:

      [user@admin ceph-ansible]$ ansible-playbook purge-docker-cluster.yml

      Copy to Clipboard Toggle word wrap

   2. To specify a different inventory file than the default one (/etc/ansible/hosts), use -i parameter:

      ansible-playbook purge-docker-cluster.yml -i inventory-file

      Copy to Clipboard Toggle word wrap

      Replace inventory-file with the path to the inventory file.

      For example:

      [user@admin ceph-ansible]$ ansible-playbook purge-docker-cluster.yml -i ~/ansible/hosts

      Copy to Clipboard Toggle word wrap

   3. To skip the removal of the Ceph container image, use the --skip-tags=”remove_img” option:

      [user@admin ceph-ansible]$ ansible-playbook --skip-tags="remove_img" purge-docker-cluster.yml

      Copy to Clipboard Toggle word wrap

   4. To skip the removal of the packages that were installed during the installation, use the --skip-tags=”with_pkg” option:

      [user@admin ceph-ansible]$ ansible-playbook --skip-tags="with_pkg" purge-docker-cluster.yml

      Copy to Clipboard Toggle word wrap