Administration Guide

1. Stop the clients from using the Block Device images RADOS Gateway - Ceph Object Gateway on this cluster and any other clients.

2. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

3. The cluster must be in healthy state (Health_OK and all PGs active+clean) before proceeding. Run ceph status on the host with the client keyrings, for example, the Ceph Monitor or OpenStack controller nodes, to ensure the cluster is healthy.

   Example

   [ceph: root@host01 /]# ceph -s

4. If you use the Ceph File System (CephFS), bring down the CephFS cluster:

   Syntax

   ceph fs set FS_NAME max_mds 1
   ceph fs fail FS_NAME
   ceph status
   ceph fs set FS_NAME joinable false

   Example

   [ceph: root@host01 /]# ceph fs set cephfs max_mds 1
   [ceph: root@host01 /]# ceph fs fail cephfs
   [ceph: root@host01 /]# ceph status
   [ceph: root@host01 /]# ceph fs set cephfs joinable false

5. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd set noout
   [ceph: root@host01 /]# ceph osd set norecover
   [ceph: root@host01 /]# ceph osd set norebalance
   [ceph: root@host01 /]# ceph osd set nobackfill
   [ceph: root@host01 /]# ceph osd set nodown
   [ceph: root@host01 /]# ceph osd set pause

   Important

   The above example is only for stopping the service and each OSD in the OSD node and it needs to be repeated on each OSD node.

6. If the MDS and Ceph Object Gateway nodes are on their own dedicated nodes, power them off.

7. Get the systemd target of the daemons:

   Example

   [root@host01 ~]# systemctl list-units --type target | grep ceph
   ceph-0b007564-ec48-11ee-b736-525400fd02f8.target loaded active active Ceph cluster 0b007564-ec48-11ee-b736-525400fd02f8
   ceph.target                                      loaded active active All Ceph clusters and services

8. Disable the target that includes the cluster FSID:

   Example

   [root@host01 ~]# systemctl disable ceph-0b007564-ec48-11ee-b736-525400fd02f8.target
   
   Removed "/etc/systemd/system/multi-user.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target".
   Removed "/etc/systemd/system/ceph.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target".

9. Stop the target:

   Example

   [root@host01 ~]# systemctl stop ceph-0b007564-ec48-11ee-b736-525400fd02f8.target

   This stops all the daemons on the host that needs to be stopped.

10. Shutdown the node:

    Example

    [root@host01 ~]# shutdown
    Shutdown scheduled for Wed 2024-03-27 11:47:19 EDT, use 'shutdown -c' to cancel.

11. Repeat the above steps for all the nodes of the cluster.

1. If network equipment was involved, ensure it is powered ON and stable prior to powering ON any Ceph hosts or nodes.
2. Power ON the administration node.

3. Enable the systemd target to get all the daemons running:

   Example

   [root@host01 ~]# systemctl enable ceph-0b007564-ec48-11ee-b736-525400fd02f8.target
   Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target → /etc/systemd/system/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target.
   Created symlink /etc/systemd/system/ceph.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target → /etc/systemd/system/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target.

4. Start the systemd target:

   Example

   [root@host01 ~]# systemctl start ceph-0b007564-ec48-11ee-b736-525400fd02f8.target

5. Wait for all the nodes to come up. Verify all the services are up and there are no connectivity issues between the nodes.

6. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd unset noout
   [ceph: root@host01 /]# ceph osd unset norecover
   [ceph: root@host01 /]# ceph osd unset norebalance
   [ceph: root@host01 /]# ceph osd unset nobackfill
   [ceph: root@host01 /]# ceph osd unset nodown
   [ceph: root@host01 /]# ceph osd unset pause

7. If you use the Ceph File System (CephFS), bring the CephFS cluster back up by setting the joinable flag to true:

   Syntax

   ceph fs set FS_NAME joinable true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true

1. Stop the clients from using the user Block Device Image and Ceph Object Gateway on this cluster and any other clients.

2. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

3. The cluster must be in healthy state (Health_OK and all PGs active+clean) before proceeding. Run ceph status on the host with the client keyrings, for example, the Ceph Monitor or OpenStack controller nodes, to ensure the cluster is healthy.

   Example

   [ceph: root@host01 /]# ceph -s

4. If you use the Ceph File System (CephFS), bring down the CephFS cluster:

   Syntax

   ceph fs set FS_NAME max_mds 1
   ceph fs fail FS_NAME
   ceph status
   ceph fs set FS_NAME joinable false
   ceph mds fail FS_NAME:N

   Example

   [ceph: root@host01 /]# ceph fs set cephfs max_mds 1
   [ceph: root@host01 /]# ceph fs fail cephfs
   [ceph: root@host01 /]# ceph status
   [ceph: root@host01 /]# ceph fs set cephfs joinable false
   [ceph: root@host01 /]# ceph mds fail cephfs:1

5. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd set noout
   [ceph: root@host01 /]# ceph osd set norecover
   [ceph: root@host01 /]# ceph osd set norebalance
   [ceph: root@host01 /]# ceph osd set nobackfill
   [ceph: root@host01 /]# ceph osd set nodown
   [ceph: root@host01 /]# ceph osd set pause

6. Stop the MDS service.

   1. Fetch the MDS service name:

      Example

      [ceph: root@host01 /]# ceph orch ls --service-type mds

   2. Stop the MDS service using the fetched name in the previous step:

      Syntax

      ceph orch stop SERVICE-NAME

7. Stop the Ceph Object Gateway services. Repeat for each deployed service.

   1. Fetch the Ceph Object Gateway service names:

      Example

      [ceph: root@host01 /]# ceph orch ls --service-type rgw

   2. Stop the Ceph Object Gateway service using the fetched name:

      Syntax

      ceph orch stop SERVICE-NAME

8. Stop the Alertmanager service:

   Example

   [ceph: root@host01 /]# ceph orch stop alertmanager

9. Stop the node-exporter service which is a part of the monitoring stack:

   Example

   [ceph: root@host01 /]# ceph orch stop node-exporter

10. Stop the Prometheus service:

    Example

    [ceph: root@host01 /]# ceph orch stop prometheus

11. Stop the Grafana dashboard service:

    Example

    [ceph: root@host01 /]# ceph orch stop grafana

12. Stop the crash service:

    Example

    [ceph: root@host01 /]# ceph orch stop crash

13. Shut down the OSD nodes from the cephadm node, one by one. Repeat this step for all the OSDs in the cluster.

    1. Fetch the OSD ID:

       Example

       [ceph: root@host01 /]# ceph orch ps --daemon-type=osd

    2. Shut down the OSD node using the OSD ID you fetched:

       Example

       [ceph: root@host01 /]# ceph orch daemon stop osd.1
       Scheduled to stop osd.1 on host 'host02'

14. Stop the monitors one by one.

    1. Identify the hosts hosting the monitors:

       Example

       [ceph: root@host01 /]# ceph orch ps --daemon-type mon

    2. On each host, stop the monitor.

       1. Identify the systemctl unit name:

          Example

          [ceph: root@host01 /]# systemctl list-units ceph-* | grep mon

       2. Stop the service:

          Syntax

          systemct stop SERVICE-NAME

15. Shut down all the hosts.

1. If network equipment was involved, ensure it is powered ON and stable prior to powering ON any Ceph hosts or nodes.
2. Power ON all the Ceph hosts.

3. Log into the administration node from the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

4. Verify all the services are in running state:

   Example

   [ceph: root@host01 /]# ceph orch ls

5. Ensure the cluster health is `Health_OK`status:

   Example

   [ceph: root@host01 /]# ceph -s

6. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd unset noout
   [ceph: root@host01 /]# ceph osd unset norecover
   [ceph: root@host01 /]# ceph osd unset norebalance
   [ceph: root@host01 /]# ceph osd unset nobackfill
   [ceph: root@host01 /]# ceph osd unset nodown
   [ceph: root@host01 /]# ceph osd unset pause

7. If you use the Ceph File System (CephFS), bring the CephFS cluster back up by setting the joinable flag to true:

   Syntax

   ceph fs set FS_NAME joinable true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true

Procedure

1. Log into the Cephadm shell:

   Example

   root@host01 ~]# cephadm shell

2. You can check on the health of the Ceph storage cluster with the following command:

   Example

   [ceph: root@host01 /]# ceph health
   HEALTH_OK

3. You can check the status of the Ceph storage cluster by running ceph status command:

   Example

   [ceph: root@host01 /]# ceph status

   The output provides the following information:

   • Cluster ID
   • Cluster health status
   • The monitor map epoch and the status of the monitor quorum.
   • The OSD map epoch and the status of OSDs.
   • The status of Ceph Managers.
   • The status of Object Gateways.
   • The placement group map version.
   • The number of placement groups and pools.
   • The notional amount of data stored and the number of objects stored.
   • The total amount of data stored.
   • The IO client operations.

   • An update on the upgrade process if the cluster is upgrading.

     Upon starting the Ceph cluster, you will likely encounter a health warning such as HEALTH_WARN XXX num placement groups stale. Wait a few moments and check it again. When the storage cluster is ready, ceph health should return a message such as HEALTH_OK. At that point, it is okay to begin using the cluster.

Procedure

1. Log into the Cephadm shell:

   Example

   root@host01 ~]# cephadm shell

2. To watch the cluster’s ongoing events, run the following command:

   Example

   [ceph: root@host01 /]# ceph -w
     cluster:
       id:     8c9b0072-67ca-11eb-af06-001a4a0002a0
       health: HEALTH_OK
   
     services:
       mon: 2 daemons, quorum Ceph5-2,Ceph5-adm (age 3d)
       mgr: Ceph5-1.nqikfh(active, since 3w), standbys: Ceph5-adm.meckej
       osd: 5 osds: 5 up (since 2d), 5 in (since 8w)
       rgw: 2 daemons active (test_realm.test_zone.Ceph5-2.bfdwcn, test_realm.test_zone.Ceph5-adm.acndrh)
   
     data:
       pools:   11 pools, 273 pgs
       objects: 459 objects, 32 KiB
       usage:   2.6 GiB used, 72 GiB / 75 GiB avail
       pgs:     273 active+clean
   
     io:
       client:   170 B/s rd, 730 KiB/s wr, 0 op/s rd, 729 op/s wr
   
   2021-06-02 15:45:21.655871 osd.0 [INF] 17.71 deep-scrub ok
   2021-06-02 15:45:47.880608 osd.1 [INF] 1.0 scrub ok
   2021-06-02 15:45:48.865375 osd.1 [INF] 1.3 scrub ok
   2021-06-02 15:45:50.866479 osd.1 [INF] 1.4 scrub ok
   2021-06-02 15:45:01.345821 mon.0 [INF] pgmap v41339: 952 pgs: 952 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:05.718640 mon.0 [INF] pgmap v41340: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:53.997726 osd.1 [INF] 1.5 scrub ok
   2021-06-02 15:45:06.734270 mon.0 [INF] pgmap v41341: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:15.722456 mon.0 [INF] pgmap v41342: 952 pgs: 952 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:46:06.836430 osd.0 [INF] 17.75 deep-scrub ok
   2021-06-02 15:45:55.720929 mon.0 [INF] pgmap v41343: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To check a storage cluster’s status, execute the following:

   Example

   [ceph: root@host01 /]# ceph status

   Or

   Example

   [ceph: root@host01 /]# ceph -s

3. In interactive mode, type ceph and press Enter:

   Example

   [ceph: root@host01 /]# ceph
   ceph> status
     cluster:
       id:     499829b4-832f-11eb-8d6d-001a4a000635
       health: HEALTH_WARN
               1 stray daemon(s) not managed by cephadm
               1/3 mons down, quorum host03,host02
               too many PGs per OSD (261 > max 250)
   
     services:
       mon:     3 daemons, quorum host03,host02 (age 3d), out of quorum: host01
       mgr:     host01.hdhzwn(active, since 9d), standbys: host05.eobuuv, host06.wquwpj
       osd:     12 osds: 11 up (since 2w), 11 in (since 5w)
       rgw:     2 daemons active (test_realm.test_zone.host04.hgbvnq, test_realm.test_zone.host05.yqqilm)
       rgw-nfs: 1 daemon active (nfs.foo.host06-rgw)
   
     data:
       pools:   8 pools, 960 pgs
       objects: 414 objects, 1.0 MiB
       usage:   5.7 GiB used, 214 GiB / 220 GiB avail
       pgs:     960 active+clean
   
     io:
       client:   41 KiB/s rd, 0 B/s wr, 41 op/s rd, 27 op/s wr
   
   ceph> health
   HEALTH_WARN 1 stray daemon(s) not managed by cephadm; 1/3 mons down, quorum host03,host02; too many PGs per OSD (261 > max 250)
   
   ceph> mon stat
   e3: 3 mons at {host01=[v2:10.74.255.0:3300/0,v1:10.74.255.0:6789/0],host02=[v2:10.74.249.253:3300/0,v1:10.74.249.253:6789/0],host03=[v2:10.74.251.164:3300/0,v1:10.74.251.164:6789/0]}, election epoch 6688, leader 1 host03, quorum 1,2 host03,host02

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To display the Ceph Monitor map, execute the following:

   Example

   [ceph: root@host01 /]# ceph mon stat

   or

   Example

   [ceph: root@host01 /]# ceph mon dump

3. To check the quorum status for the storage cluster, execute the following:

   [ceph: root@host01 /]# ceph quorum_status -f json-pretty

   Ceph returns the quorum status.

   Example

   {
       "election_epoch": 6686,
       "quorum": [
           0,
           1,
           2
       ],
       "quorum_names": [
           "host01",
           "host03",
           "host02"
       ],
       "quorum_leader_name": "host01",
       "quorum_age": 424884,
       "features": {
           "quorum_con": "4540138297136906239",
           "quorum_mon": [
               "kraken",
               "luminous",
               "mimic",
               "osdmap-prune",
               "nautilus",
               "octopus",
               "pacific",
               "elector-pinging"
           ]
       },
       "monmap": {
           "epoch": 3,
           "fsid": "499829b4-832f-11eb-8d6d-001a4a000635",
           "modified": "2021-03-15T04:51:38.621737Z",
           "created": "2021-03-12T12:35:16.911339Z",
           "min_mon_release": 16,
           "min_mon_release_name": "pacific",
           "election_strategy": 1,
           "disallowed_leaders: ": "",
           "stretch_mode": false,
           "features": {
               "persistent": [
                   "kraken",
                   "luminous",
                   "mimic",
                   "osdmap-prune",
                   "nautilus",
                   "octopus",
                   "pacific",
                   "elector-pinging"
               ],
               "optional": []
           },
           "mons": [
               {
                   "rank": 0,
                   "name": "host01",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.255.0:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.255.0:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.255.0:6789/0",
                   "public_addr": "10.74.255.0:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               },
               {
                   "rank": 1,
                   "name": "host03",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.251.164:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.251.164:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.251.164:6789/0",
                   "public_addr": "10.74.251.164:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               },
               {
                   "rank": 2,
                   "name": "host02",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.249.253:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.249.253:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.249.253:6789/0",
                   "public_addr": "10.74.249.253:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               }
           ]
       }
   }

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To use the socket:

   Syntax

   ceph daemon MONITOR_ID COMMAND

   Replace:

   • MONITOR_ID of the daemon

   • COMMAND with the command to run. Use help to list the available commands for a given daemon.

     To view the status of a Ceph Monitor:

     Example

     [ceph: root@host01 /]# ceph daemon mon.host01 help
     {
         "add_bootstrap_peer_hint": "add peer address as potential bootstrap peer for cluster bringup",
         "add_bootstrap_peer_hintv": "add peer address vector as potential bootstrap peer for cluster bringup",
         "compact": "cause compaction of monitor's leveldb/rocksdb storage",
         "config diff": "dump diff of current config and default config",
         "config diff get": "dump diff get <field>: dump diff of current and default config setting <field>",
         "config get": "config get <field>: get the config value",
         "config help": "get config setting schema and descriptions",
         "config set": "config set <field> <val> [<val> ...]: set a config variable",
         "config show": "dump current config settings",
         "config unset": "config unset <field>: unset a config variable",
         "connection scores dump": "show the scores used in connectivity-based elections",
         "connection scores reset": "reset the scores used in connectivity-based elections",
         "counter dump": "dump all labeled and non-labeled counters and their values",
         "counter schema": "dump all labeled and non-labeled counters schemas",
         "dump_historic_ops": "show recent ops",
         "dump_historic_slow_ops": "show recent slow ops",
         "dump_mempools": "get mempool stats",
         "get_command_descriptions": "list available commands",
         "git_version": "get git sha1",
         "heap": "show heap usage info (available only if compiled with tcmalloc)",
         "help": "list available commands",
         "injectargs": "inject configuration arguments into running daemon",
         "log dump": "dump recent log entries to log file",
         "log flush": "flush log entries to log file",
         "log reopen": "reopen log file",
         "mon_status": "report status of monitors",
         "ops": "show the ops currently in flight",
         "perf dump": "dump non-labeled counters and their values",
         "perf histogram dump": "dump perf histogram values",
         "perf histogram schema": "dump perf histogram schema",
         "perf reset": "perf reset <name>: perf reset all or one perfcounter name",
         "perf schema": "dump non-labeled counters schemas",
         "quorum enter": "force monitor back into quorum",
         "quorum exit": "force monitor out of the quorum",
         "sessions": "list existing sessions",
         "smart": "Query health metrics for underlying device",
         "sync_force": "force sync of and clear monitor store",
         "version": "get ceph version"
     }

     Example

     [ceph: root@host01 /]# ceph daemon mon.host01 mon_status
     
     {
         "name": "host01",
         "rank": 0,
         "state": "leader",
         "election_epoch": 120,
         "quorum": [
             0,
             1,
             2
         ],
         "quorum_age": 206358,
         "features": {
             "required_con": "2449958747317026820",
             "required_mon": [
                 "kraken",
                 "luminous",
                 "mimic",
                 "osdmap-prune",
                 "nautilus",
                 "octopus",
                 "pacific",
                 "elector-pinging"
             ],
             "quorum_con": "4540138297136906239",
             "quorum_mon": [
                 "kraken",
                 "luminous",
                 "mimic",
                 "osdmap-prune",
                 "nautilus",
                 "octopus",
                 "pacific",
                 "elector-pinging"
             ]
         },
         "outside_quorum": [],
         "extra_probe_peers": [],
         "sync_provider": [],
         "monmap": {
             "epoch": 3,
             "fsid": "81a4597a-b711-11eb-8cb8-001a4a000740",
             "modified": "2021-05-18T05:50:17.782128Z",
             "created": "2021-05-17T13:13:13.383313Z",
             "min_mon_release": 16,
             "min_mon_release_name": "pacific",
             "election_strategy": 1,
             "disallowed_leaders: ": "",
             "stretch_mode": false,
             "features": {
                 "persistent": [
                     "kraken",
                     "luminous",
                     "mimic",
                     "osdmap-prune",
                     "nautilus",
                     "octopus",
                     "pacific",
                     "elector-pinging"
                 ],
                 "optional": []
             },
             "mons": [
                 {
                     "rank": 0,
                     "name": "host01",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.41:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.41:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.41:6789/0",
                     "public_addr": "10.74.249.41:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 },
                 {
                     "rank": 1,
                     "name": "host02",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.55:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.55:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.55:6789/0",
                     "public_addr": "10.74.249.55:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 },
                 {
                     "rank": 2,
                     "name": "host03",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.49:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.49:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.49:6789/0",
                     "public_addr": "10.74.249.49:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 }
             ]
         },
         "feature_map": {
             "mon": [
                 {
                     "features": "0x3f01cfb9fffdffff",
                     "release": "luminous",
                     "num": 1
                 }
             ],
             "osd": [
                 {
                     "features": "0x3f01cfb9fffdffff",
                     "release": "luminous",
                     "num": 3
                 }
             ]
         },
         "stretch_mode": false
     }

3. Alternatively, specify the Ceph daemon by using its socket file:

   Syntax

   ceph daemon /var/run/ceph/SOCKET_FILE COMMAND

4. To view the status of a Ceph OSD named osd.0 on the specific host:

   Example

   [ceph: root@host01 /]# ceph daemon /var/run/ceph/ceph-osd.0.asok status
   {
       "cluster_fsid": "9029b252-1668-11ee-9399-001a4a000429",
       "osd_fsid": "1de9b064-b7a5-4c54-9395-02ccda637d21",
       "whoami": 0,
       "state": "active",
       "oldest_map": 1,
       "newest_map": 58,
       "num_pgs": 33
   }

   Note

   You can use help instead of status for the various options that are available for the specific daemon.

5. To list all socket files for the Ceph processes:

   Example

   [ceph: root@host01 /]# ls /var/run/ceph

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To retrieve a list of placement groups:

   Example

   [ceph: root@host01 /]# ceph pg dump

3. View which Ceph OSDs are in the Acting Set or in the Up Set for a given placement group:

   Syntax

   ceph pg map PG_NUM

   Example

   [ceph: root@host01 /]# ceph pg map 128

   Note

   If the Up Set and Acting Set do not match, this may be an indicator that the storage cluster rebalancing itself or of a potential problem with the storage cluster.

Procedure

1. To identify stuck placement groups, execute the following:

   Syntax

   ceph pg dump_stuck {inactive|unclean|stale|undersized|degraded [inactive|unclean|stale|undersized|degraded...]} {<int>}

   Example

   [ceph: root@host01 /]# ceph pg dump_stuck stale
   OK

Procedure

1. To find the object location, all you need is the object name and the pool name:

   Syntax

   ceph osd map POOL_NAME OBJECT_NAME

   Example

   [ceph: root@host01 /]# ceph osd map mypool myobject

Procedure

1. Set the location of each monitor, matching your CRUSH map:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph mon set_location host01 datacenter=DC1
   [ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC1
   [ceph: root@host01 /]# ceph mon set_location host04 datacenter=DC2
   [ceph: root@host01 /]# ceph mon set_location host05 datacenter=DC2
   [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3

2. Generate a CRUSH rule which places two copies on each data center:

   Syntax

   ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
   crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
   [ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

   1. Edit the decompiled CRUSH map file to add a new rule:

      Example

      rule stretch_rule {
              id 1 

      1

      
              type replicated
              min_size 1
              max_size 10
              step take DC1 

      2

      
              step chooseleaf firstn 2 type host
              step emit
              step take DC2 

      3

      
              step chooseleaf firstn 2 type host
              step emit
      }

      1

      The rule id has to be unique. In this example, there is only one more rule with id 0, thereby the id 1 is used, however you might need to use a different rule ID depending on the number of existing rules.

      2 3

      In this example, there are two data center buckets named DC1 and DC2.

      Note

      This rule makes the cluster have read-affinity towards data center DC1. Therefore, all the reads or writes happen through Ceph OSDs placed in DC1.

      If this is not desirable, and reads or writes are to be distributed evenly across the zones, the CRUSH rule is the following:

      Example

      rule stretch_rule {
      id 1
      type replicated
      min_size 1
      max_size 10
      step take default
      step choose firstn 0 type datacenter
      step chooseleaf firstn 2 type host
      step emit
      }

      In this rule, the data center is selected randomly and automatically.

      See CRUSH rules for more information on firstn and indep options.

3. Inject the CRUSH map to make the rule available to the cluster:

   Syntax

   crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
   ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
   [ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

4. If you do not run the monitors in connectivity mode, set the election strategy to connectivity:

   Example

   [ceph: root@host01 /]# ceph mon set election_strategy connectivity

5. Enter stretch mode by setting the location of the tiebreaker monitor to split across the data centers:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER
   ceph mon enable_stretch_mode HOST stretch_rule datacenter

   Example

   [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3
   [ceph: root@host01 /]# ceph mon enable_stretch_mode host07 stretch_rule datacenter

   In this example the monitor mon.host07 is the tiebreaker.

   Important

   The location of the tiebreaker monitor should differ from the data centers to which you previously set the non-tiebreaker monitors. In the example above, it is data center DC3.

   Important

   Do not add this data center to the CRUSH map as it results in the following error when you try to enter stretch mode:

   Error EINVAL: there are 3 datacenters in the cluster but stretch mode currently only works with 2!

   Note

   If you are writing your own tooling for deploying Ceph, you can use a new --set-crush-location option when booting monitors, instead of running the ceph mon set_location command. This option accepts only a single bucket=location pair, for example ceph-mon --set-crush-location 'datacenter=DC1', which must match the bucket type you specified when running the enable_stretch_mode command.

6. Verify that the stretch mode is enabled successfully:

   Example

   [ceph: root@host01 /]# ceph osd dump
   
   epoch 361
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   created 2023-01-16T05:47:28.482717+0000
   modified 2023-01-17T17:36:50.066183+0000
   flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
   crush_version 31
   full_ratio 0.95
   backfillfull_ratio 0.92
   nearfull_ratio 0.85
   require_min_compat_client luminous
   min_compat_client luminous
   require_osd_release quincy
   stretch_mode_enabled true
   stretch_bucket_count 2
   degraded_stretch_mode 0
   recovering_stretch_mode 0
   stretch_mode_bucket 8

   The stretch_mode_enabled should be set to true. You can also see the number of stretch buckets, stretch mode buckets, and if the stretch mode is degraded or recovering.

7. Verify that the monitors are in an appropriate locations:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host07
   disallowed_leaders host07
   0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host07; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
   dumped monmap epoch 19

   You can also see which monitor is the tiebreaker, and the monitor election strategy.

Procedure

1. Create a CRUSH rule that makes use of this OSD crush topology by installing the ceph-base RPM package in order to use the crushtool command.

   Syntax

   dnf -y install ceph-base

2. Get the compiled CRUSH map from the cluster.

   Syntax

   ceph osd getcrushmap > /etc/ceph/crushmap.bin

3. Decompile the CRUSH map and convert it to a text file to edit it.

   Syntax

   crushtool -d /etc/ceph/crushmap.bin -o /etc/ceph/crushmap.txt

4. Add the following rule to the CRUSH map by editing the /etc/ceph/crushmap.txt at the end of the file. This rule distributes reads and writes evenly across the data center.

   Syntax

   rule stretch_rule {
           id 1
           type replicated
           step take default
           step choose firstn 0 type datacenter
           step chooseleaf firstn 2 type host
           step emit
    }

   1. Optionally have the cluster with a read/write affinity towards data center 1.

      Syntax

      rule stretch_rule {
               id 1
               type replicated
               step take DC1
               step chooseleaf firstn 2 type host
               step emit
               step take DC2
               step chooseleaf firstn 2 type host
               step emit
       }

      The CRUSH rule declared contains the following information:
           Rule name
                Description: A unique name for identifying the rule.
                Value: stretch_rule
           id
                Description: A unique whole number for identifying the rule.
                Value: 1
           type
                Description: Describes a rule for either a storage drive replicated or erasure-coded.
                Value: replicated
           step take default
                Description: Takes the root bucket called default, and begins iterating down the tree.
           step take DC1
                Description: Takes the bucket called DC1, and begins iterating down the tree.
           step choose firstn 0 type datacenter
                Description: Selects the datacenter bucket, and goes into its subtrees.
           step chooseleaf firstn 2 type host
                Description: Selects the number of buckets of the given type. In this case, it is two different hosts located in the datacenter it entered at the previous level.
           step emit
                Description: Outputs the current value and empties the stack. Typically used at the end of a rule, but may also be used to pick from different trees in the same rule.

5. Compile the new CRUSH map from /etc/ceph/crushmap.txt and convert it to a binary file /etc/ceph/crushmap2.bin.

   Syntax

   crushtool -c /path/to/crushmap.txt -o /path/to/crushmap2.bin

   Example

   [ceph: root@host01 /]# crushtool -c /etc/ceph/crushmap.txt -o /etc/ceph/crushmap2.bin

6. Inject the newly created CRUSH map back into the cluster.

   Syntax

   ceph osd setcrushmap -i /path/to/compiled_crushmap

   Example

   [ceph: root@host01 /]# ceph osd setcrushmap -i /path/to/compiled_crushmap
   17

   Note

   The number 17 is a counter and increases (18,19, and so on) depending on the changes that are made to the CRUSH map.

Procedure

1. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOST_1 HOST_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls

2. Deploy the OSDs on specific hosts or on all the available devices:

   • Create an OSD from a specific device on a specific host:

     Syntax

     ceph orch daemon add osd HOST:DEVICE_PATH

     Example

     [ceph: root@host01 /]# ceph orch daemon add osd host03:/dev/sdb

   • Deploy OSDs on any available and unused devices:

     Important

     This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

3. Move the OSD hosts under the CRUSH bucket:

   Syntax

   ceph osd crush move HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph osd crush move host03 datacenter=DC1
   [ceph: root@host01 /]# ceph osd crush move host06 datacenter=DC2

   Note

   Ensure you add the same topology nodes on both sites. Issues might arise if hosts are added only on one site.

Procedure

1. Disable automated monitor deployment:

   Example

   [ceph: root@host01 /]# ceph orch apply mon --unmanaged
   
   Scheduled mon update…

2. View the monitors in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 5 daemons, quorum host01, host02, host04, host05 (age 30s), out of quorum: host07

3. Set the monitor in quorum as a new tiebreaker:

   Syntax

   ceph mon set_new_tiebreaker NEW_HOST

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host02

   Important

   You get an error message if the monitor is in the same location as existing non-tiebreaker monitors:

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host02
   
   Error EINVAL: mon.host02 has location DC1, which matches mons host02 on the datacenter dividing bucket for stretch mode.

   If that happens, change the location of the monitor:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC3

4. Remove the failed tiebreaker monitor:

   Syntax

   ceph orch daemon rm FAILED_TIEBREAKER_MONITOR --force

   Example

   [ceph: root@host01 /]# ceph orch daemon rm mon.host07 --force
   
   Removed mon.host07 from host 'host07'

5. Once the monitor is removed from the host, redeploy the monitor:

   Syntax

   ceph mon add HOST IP_ADDRESS datacenter=DATACENTER
   ceph orch daemon add mon HOST

   Example

   [ceph: root@host01 /]# ceph mon add host07 213.222.226.50 datacenter=DC1
   [ceph: root@host01 /]# ceph orch daemon add mon host07

6. Ensure there are five monitors in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 5 daemons, quorum host01, host02, host04, host05, host07 (age 15s)

7. Verify that everything is configured properly:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host02
   disallowed_leaders host02
   0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host02; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host07; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host03; crush_location {datacenter=DC2}
   dumped monmap epoch 19

8. Redeploy the monitors:

   Syntax

   ceph orch apply mon --placement="HOST_1, HOST_2, HOST_3, HOST_4, HOST_5”

   Example

   [ceph: root@host01 /]# ceph orch apply mon --placement="host01, host02, host04, host05, host07"
   
   Scheduled mon update...

Procedure

1. Add a new monitor to the cluster:

   1. Manually add the crush_location to the new monitor:

      Syntax

      ceph mon add NEW_HOST IP_ADDRESS datacenter=DATACENTER

      Example

      [ceph: root@host01 /]# ceph mon add host06 213.222.226.50 datacenter=DC3
      
      adding mon.host06 at [v2:213.222.226.50:3300/0,v1:213.222.226.50:6789/0]

      Note

      The new monitor has to be in a different location than existing non-tiebreaker monitors.

   2. Disable automated monitor deployment:

      Example

      [ceph: root@host01 /]# ceph orch apply mon --unmanaged
      
      Scheduled mon update…

   3. Deploy the new monitor:

      Syntax

      ceph orch daemon add mon NEW_HOST

      Example

      [ceph: root@host01 /]# ceph orch daemon add mon host06

2. Ensure there are 6 monitors, from which 5 are in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 6 daemons, quorum host01, host02, host04, host05, host06 (age 30s), out of quorum: host07

3. Set the new monitor as a new tiebreaker:

   Syntax

   ceph mon set_new_tiebreaker NEW_HOST

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host06

4. Remove the failed tiebreaker monitor:

   Syntax

   ceph orch daemon rm FAILED_TIEBREAKER_MONITOR --force

   Example

   [ceph: root@host01 /]# ceph orch daemon rm mon.host07 --force
   
   Removed mon.host07 from host 'host07'

5. Verify that everything is configured properly:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host06
   disallowed_leaders host06
   0: [v2:213.222.226.50:3300/0,v1:213.222.226.50:6789/0] mon.host06; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
   dumped monmap epoch 19

6. Redeploy the monitors:

   Syntax

   ceph orch apply mon --placement="HOST_1, HOST_2, HOST_3, HOST_4, HOST_5”

   Example

   [ceph: root@host01 /]# ceph orch apply mon --placement="host01, host02, host04, host05, host06"
   
   Scheduled mon update…

Procedure

1. To list the users in the storage cluster, execute the following:

   Example

   [ceph: root@host01 /]# ceph auth list
   installed auth entries:
   
   osd.10
   	key: AQBW7U5gqOsEExAAg/CxSwZ/gSh8iOsDV3iQOA==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   osd.11
   	key: AQBX7U5gtj/JIhAAPsLBNG+SfC2eMVEFkl3vfA==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   osd.9
   	key: AQBV7U5g1XDULhAAKo2tw6ZhH1jki5aVui2v7g==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   client.admin
   	key: AQADYEtgFfD3ExAAwH+C1qO7MSLE4TWRfD2g6g==
   	caps: [mds] allow *
   	caps: [mgr] allow *
   	caps: [mon] allow *
   	caps: [osd] allow *
   client.bootstrap-mds
   	key: AQAHYEtgpbkANBAANqoFlvzEXFwD8oB0w3TF4Q==
   	caps: [mon] allow profile bootstrap-mds
   client.bootstrap-mgr
   	key: AQAHYEtg3dcANBAAVQf6brq3sxTSrCrPe0pKVQ==
   	caps: [mon] allow profile bootstrap-mgr
   client.bootstrap-osd
   	key: AQAHYEtgD/QANBAATS9DuP3DbxEl86MTyKEmdw==
   	caps: [mon] allow profile bootstrap-osd
   client.bootstrap-rbd
   	key: AQAHYEtgjxEBNBAANho25V9tWNNvIKnHknW59A==
   	caps: [mon] allow profile bootstrap-rbd
   client.bootstrap-rbd-mirror
   	key: AQAHYEtgdE8BNBAAr6rLYxZci0b2hoIgH9GXYw==
   	caps: [mon] allow profile bootstrap-rbd-mirror
   client.bootstrap-rgw
   	key: AQAHYEtgwGkBNBAAuRzI4WSrnowBhZxr2XtTFg==
   	caps: [mon] allow profile bootstrap-rgw
   client.crash.host04
   	key: AQCQYEtgz8lGGhAAy5bJS8VH9fMdxuAZ3CqX5Q==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.crash.host02
   	key: AQDuYUtgqgfdOhAAsyX+Mo35M+HFpURGad7nJA==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.crash.host03
   	key: AQB98E5g5jHZAxAAklWSvmDsh2JaL5G7FvMrrA==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.nfs.foo.host03
   	key: AQCgTk9gm+HvMxAAHbjG+XpdwL6prM/uMcdPdQ==
   	caps: [mon] allow r
   	caps: [osd] allow rw pool=nfs-ganesha namespace=foo
   client.nfs.foo.host03-rgw
   	key: AQCgTk9g8sJQNhAAPykcoYUuPc7IjubaFx09HQ==
   	caps: [mon] allow r
   	caps: [osd] allow rwx tag rgw *=*
   client.rgw.test_realm.test_zone.host01.hgbvnq
   	key: AQD5RE9gAQKdCRAAJzxDwD/dJObbInp9J95sXw==
   	caps: [mgr] allow rw
   	caps: [mon] allow *
   	caps: [osd] allow rwx tag rgw *=*
   client.rgw.test_realm.test_zone.host02.yqqilm
   	key: AQD0RE9gkxA4ExAAFXp3pLJWdIhsyTe2ZR6Ilw==
   	caps: [mgr] allow rw
   	caps: [mon] allow *
   	caps: [osd] allow rwx tag rgw *=*
   mgr.host01.hdhzwn
   	key: AQAEYEtg3lhIBxAAmHodoIpdvnxK0llWF80ltQ==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *
   mgr.host02.eobuuv
   	key: AQAn6U5gzUuiABAA2Fed+jPM1xwb4XDYtrQxaQ==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *
   mgr.host03.wquwpj
   	key: AQAd6U5gIzWsLBAAbOKUKZlUcAVe9kBLfajMKw==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *

Procedure

1. To retrieve a specific user, key and capabilities, execute the following:

   Syntax

   ceph auth export TYPE.ID

   Example

   [ceph: root@host01 /]# ceph auth export mgr.host02.eobuuv

2. You can also use the -o FILE_NAME option.

   Syntax

   ceph auth export TYPE.ID -o FILE_NAME

   Example

   [ceph: root@host01 /]# ceph auth export osd.9 -o filename
   export auth(key=AQBV7U5g1XDULhAAKo2tw6ZhH1jki5aVui2v7g==)

Procedure

1. To add capabilities, use the form:

   Syntax

   ceph auth caps USERTYPE.USERID DAEMON 'allow [r|w|x|*|...] [pool=POOL_NAME] [namespace=NAMESPACE_NAME]'

   Example

   [ceph: root@host01 /]# ceph auth caps client.john mon 'allow r' osd 'allow rw pool=mypool'
   [ceph: root@host01 /]# ceph auth caps client.paul mon 'allow rw' osd 'allow rwx pool=mypool'
   [ceph: root@host01 /]# ceph auth caps client.brian-manager mon 'allow *' osd 'allow *'

2. To remove a capability, you may reset the capability. If you want the user to have no access to a particular daemon that was previously set, specify an empty string:

   Example

   [ceph: root@host01 /]# ceph auth caps client.ringo mon ' ' osd ' '

Procedure

1. To delete a user, use ceph auth del:

   Syntax

   ceph auth del TYPE.ID

   Example

   [ceph: root@host01 /]# ceph auth del osd.6