Administration Guide

1. Stop the clients from using the Block Device images RADOS Gateway - Ceph Object Gateway on this cluster and any other clients.

2. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

3. The cluster must be in healthy state (Health_OK and all PGs active+clean) before proceeding. Run ceph status on the host with the client keyrings, for example, the Ceph Monitor or OpenStack controller nodes, to ensure the cluster is healthy.

   Example

   [ceph: root@host01 /]# ceph -s

4. If you use the Ceph File System (CephFS), bring down the CephFS cluster:

   Syntax

   ceph fs set FS_NAME max_mds 1
   ceph fs fail FS_NAME
   ceph status
   ceph fs set FS_NAME joinable false

   Example

   [ceph: root@host01 /]# ceph fs set cephfs max_mds 1
   [ceph: root@host01 /]# ceph fs fail cephfs
   [ceph: root@host01 /]# ceph status
   [ceph: root@host01 /]# ceph fs set cephfs joinable false

5. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd set noout
   [ceph: root@host01 /]# ceph osd set norecover
   [ceph: root@host01 /]# ceph osd set norebalance
   [ceph: root@host01 /]# ceph osd set nobackfill
   [ceph: root@host01 /]# ceph osd set nodown
   [ceph: root@host01 /]# ceph osd set pause

   Important

   The above example is only for stopping the service and each OSD in the OSD node and it needs to be repeated on each OSD node.

6. If the MDS and Ceph Object Gateway nodes are on their own dedicated nodes, power them off.

7. Get the systemd target of the daemons:

   Example

   [root@host01 ~]# systemctl list-units --type target | grep ceph
   ceph-0b007564-ec48-11ee-b736-525400fd02f8.target loaded active active Ceph cluster 0b007564-ec48-11ee-b736-525400fd02f8
   ceph.target                                      loaded active active All Ceph clusters and services

8. Disable the target that includes the cluster FSID:

   Example

   [root@host01 ~]# systemctl disable ceph-0b007564-ec48-11ee-b736-525400fd02f8.target
   
   Removed "/etc/systemd/system/multi-user.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target".
   Removed "/etc/systemd/system/ceph.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target".

9. Stop the target:

   Example

   [root@host01 ~]# systemctl stop ceph-0b007564-ec48-11ee-b736-525400fd02f8.target

   This stops all the daemons on the host that needs to be stopped.

10. Shutdown the node:

    Example

    [root@host01 ~]# shutdown
    Shutdown scheduled for Wed 2024-03-27 11:47:19 EDT, use 'shutdown -c' to cancel.

11. Repeat the above steps for all the nodes of the cluster.

1. If network equipment was involved, ensure it is powered ON and stable prior to powering ON any Ceph hosts or nodes.
2. Power ON the administration node.

3. Enable the systemd target to get all the daemons running:

   Example

   [root@host01 ~]# systemctl enable ceph-0b007564-ec48-11ee-b736-525400fd02f8.target
   Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target → /etc/systemd/system/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target.
   Created symlink /etc/systemd/system/ceph.target.wants/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target → /etc/systemd/system/ceph-0b007564-ec48-11ee-b736-525400fd02f8.target.

4. Start the systemd target:

   Example

   [root@host01 ~]# systemctl start ceph-0b007564-ec48-11ee-b736-525400fd02f8.target

5. Wait for all the nodes to come up. Verify all the services are up and there are no connectivity issues between the nodes.

6. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd unset noout
   [ceph: root@host01 /]# ceph osd unset norecover
   [ceph: root@host01 /]# ceph osd unset norebalance
   [ceph: root@host01 /]# ceph osd unset nobackfill
   [ceph: root@host01 /]# ceph osd unset nodown
   [ceph: root@host01 /]# ceph osd unset pause

7. If you use the Ceph File System (CephFS), bring the CephFS cluster back up by setting the joinable flag to true:

   Syntax

   ceph fs set FS_NAME joinable true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true

1. Stop the clients from using the user Block Device Image and Ceph Object Gateway on this cluster and any other clients.

2. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

3. The cluster must be in healthy state (Health_OK and all PGs active+clean) before proceeding. Run ceph status on the host with the client keyrings, for example, the Ceph Monitor or OpenStack controller nodes, to ensure the cluster is healthy.

   Example

   [ceph: root@host01 /]# ceph -s

4. If you use the Ceph File System (CephFS), bring down the CephFS cluster:

   Syntax

   ceph fs set FS_NAME max_mds 1
   ceph fs fail FS_NAME
   ceph status
   ceph fs set FS_NAME joinable false
   ceph mds fail FS_NAME:N

   Example

   [ceph: root@host01 /]# ceph fs set cephfs max_mds 1
   [ceph: root@host01 /]# ceph fs fail cephfs
   [ceph: root@host01 /]# ceph status
   [ceph: root@host01 /]# ceph fs set cephfs joinable false
   [ceph: root@host01 /]# ceph mds fail cephfs:1

5. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd set noout
   [ceph: root@host01 /]# ceph osd set norecover
   [ceph: root@host01 /]# ceph osd set norebalance
   [ceph: root@host01 /]# ceph osd set nobackfill
   [ceph: root@host01 /]# ceph osd set nodown
   [ceph: root@host01 /]# ceph osd set pause

6. Stop the MDS service.

   1. Fetch the MDS service name:

      Example

      [ceph: root@host01 /]# ceph orch ls --service-type mds

   2. Stop the MDS service using the fetched name in the previous step:

      Syntax

      ceph orch stop SERVICE-NAME

7. Stop the Ceph Object Gateway services. Repeat for each deployed service.

   1. Fetch the Ceph Object Gateway service names:

      Example

      [ceph: root@host01 /]# ceph orch ls --service-type rgw

   2. Stop the Ceph Object Gateway service using the fetched name:

      Syntax

      ceph orch stop SERVICE-NAME

8. Stop the Alertmanager service:

   Example

   [ceph: root@host01 /]# ceph orch stop alertmanager

9. Stop the node-exporter service which is a part of the monitoring stack:

   Example

   [ceph: root@host01 /]# ceph orch stop node-exporter

10. Stop the Prometheus service:

    Example

    [ceph: root@host01 /]# ceph orch stop prometheus

11. Stop the Grafana dashboard service:

    Example

    [ceph: root@host01 /]# ceph orch stop grafana

12. Stop the crash service:

    Example

    [ceph: root@host01 /]# ceph orch stop crash

13. Shut down the OSD nodes from the cephadm node, one by one. Repeat this step for all the OSDs in the cluster.

    1. Fetch the OSD ID:

       Example

       [ceph: root@host01 /]# ceph orch ps --daemon-type=osd

    2. Shut down the OSD node using the OSD ID you fetched:

       Example

       [ceph: root@host01 /]# ceph orch daemon stop osd.1
       Scheduled to stop osd.1 on host 'host02'

14. Stop the monitors one by one.

    1. Identify the hosts hosting the monitors:

       Example

       [ceph: root@host01 /]# ceph orch ps --daemon-type mon

    2. On each host, stop the monitor.

       1. Identify the systemctl unit name:

          Example

          [ceph: root@host01 /]# systemctl list-units ceph-* | grep mon

       2. Stop the service:

          Syntax

          systemct stop SERVICE-NAME

15. Shut down all the hosts.

1. If network equipment was involved, ensure it is powered ON and stable prior to powering ON any Ceph hosts or nodes.
2. Power ON all the Ceph hosts.

3. Log into the administration node from the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

4. Verify all the services are in running state:

   Example

   [ceph: root@host01 /]# ceph orch ls

5. Ensure the cluster health is `Health_OK`status:

   Example

   [ceph: root@host01 /]# ceph -s

6. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags. Run the following on a node with the client keyrings, for example, the Ceph Monitor or OpenStack controller node:

   Example

   [ceph: root@host01 /]# ceph osd unset noout
   [ceph: root@host01 /]# ceph osd unset norecover
   [ceph: root@host01 /]# ceph osd unset norebalance
   [ceph: root@host01 /]# ceph osd unset nobackfill
   [ceph: root@host01 /]# ceph osd unset nodown
   [ceph: root@host01 /]# ceph osd unset pause

7. If you use the Ceph File System (CephFS), bring the CephFS cluster back up by setting the joinable flag to true:

   Syntax

   ceph fs set FS_NAME joinable true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true

Procedure

1. To run the ceph utility in interactive mode.

   Syntax

   podman exec -it ceph-mon-MONITOR_NAME /bin/bash

   Replace

   • MONITOR_NAME with the name of the Ceph Monitor container, found by running the podman ps command.

   Example

   [root@host01 ~]# podman exec -it ceph-499829b4-832f-11eb-8d6d-001a4a000635-mon.host01 /bin/bash

Procedure

1. Log into the Cephadm shell:

   Example

   root@host01 ~]# cephadm shell

2. You can check on the health of the Ceph storage cluster with the following command:

   Example

   [ceph: root@host01 /]# ceph health
   HEALTH_OK

3. You can check the status of the Ceph storage cluster by running ceph status command:

   Example

   [ceph: root@host01 /]# ceph status

   The output provides the following information:

   • Cluster ID
   • Cluster health status
   • The monitor map epoch and the status of the monitor quorum.
   • The OSD map epoch and the status of OSDs.
   • The status of Ceph Managers.
   • The status of Object Gateways.
   • The placement group map version.
   • The number of placement groups and pools.
   • The notional amount of data stored and the number of objects stored.

   • The total amount of data stored.

     Upon starting the Ceph cluster, you will likely encounter a health warning such as HEALTH_WARN XXX num placement groups stale. Wait a few moments and check it again. When the storage cluster is ready, ceph health should return a message such as HEALTH_OK. At that point, it is okay to begin using the cluster.

Procedure

1. Log into the Cephadm shell:

   Example

   root@host01 ~]# cephadm shell

2. To watch the cluster’s ongoing events, run the following command:

   Example

   [ceph: root@host01 /]# ceph -w
     cluster:
       id:     8c9b0072-67ca-11eb-af06-001a4a0002a0
       health: HEALTH_OK
   
     services:
       mon: 2 daemons, quorum Ceph5-2,Ceph5-adm (age 3d)
       mgr: Ceph5-1.nqikfh(active, since 3w), standbys: Ceph5-adm.meckej
       osd: 5 osds: 5 up (since 2d), 5 in (since 8w)
       rgw: 2 daemons active (test_realm.test_zone.Ceph5-2.bfdwcn, test_realm.test_zone.Ceph5-adm.acndrh)
   
     data:
       pools:   11 pools, 273 pgs
       objects: 459 objects, 32 KiB
       usage:   2.6 GiB used, 72 GiB / 75 GiB avail
       pgs:     273 active+clean
   
     io:
       client:   170 B/s rd, 730 KiB/s wr, 0 op/s rd, 729 op/s wr
   
   2021-06-02 15:45:21.655871 osd.0 [INF] 17.71 deep-scrub ok
   2021-06-02 15:45:47.880608 osd.1 [INF] 1.0 scrub ok
   2021-06-02 15:45:48.865375 osd.1 [INF] 1.3 scrub ok
   2021-06-02 15:45:50.866479 osd.1 [INF] 1.4 scrub ok
   2021-06-02 15:45:01.345821 mon.0 [INF] pgmap v41339: 952 pgs: 952 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:05.718640 mon.0 [INF] pgmap v41340: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:53.997726 osd.1 [INF] 1.5 scrub ok
   2021-06-02 15:45:06.734270 mon.0 [INF] pgmap v41341: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:45:15.722456 mon.0 [INF] pgmap v41342: 952 pgs: 952 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail
   2021-06-02 15:46:06.836430 osd.0 [INF] 17.75 deep-scrub ok
   2021-06-02 15:45:55.720929 mon.0 [INF] pgmap v41343: 952 pgs: 1 active+clean+scrubbing+deep, 951 active+clean; 17130 MB data, 115 GB used, 167 GB / 297 GB avail

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To check a storage cluster’s status, execute the following:

   Example

   [ceph: root@host01 /]# ceph status

   Or

   Example

   [ceph: root@host01 /]# ceph -s

3. In interactive mode, type ceph and press Enter:

   Example

   [ceph: root@host01 /]# ceph
   ceph> status
     cluster:
       id:     499829b4-832f-11eb-8d6d-001a4a000635
       health: HEALTH_WARN
               1 stray daemon(s) not managed by cephadm
               1/3 mons down, quorum host03,host02
               too many PGs per OSD (261 > max 250)
   
     services:
       mon:     3 daemons, quorum host03,host02 (age 3d), out of quorum: host01
       mgr:     host01.hdhzwn(active, since 9d), standbys: host05.eobuuv, host06.wquwpj
       osd:     12 osds: 11 up (since 2w), 11 in (since 5w)
       rgw:     2 daemons active (test_realm.test_zone.host04.hgbvnq, test_realm.test_zone.host05.yqqilm)
       rgw-nfs: 1 daemon active (nfs.foo.host06-rgw)
   
     data:
       pools:   8 pools, 960 pgs
       objects: 414 objects, 1.0 MiB
       usage:   5.7 GiB used, 214 GiB / 220 GiB avail
       pgs:     960 active+clean
   
     io:
       client:   41 KiB/s rd, 0 B/s wr, 41 op/s rd, 27 op/s wr
   
   ceph> health
   HEALTH_WARN 1 stray daemon(s) not managed by cephadm; 1/3 mons down, quorum host03,host02; too many PGs per OSD (261 > max 250)
   
   ceph> mon stat
   e3: 3 mons at {host01=[v2:10.74.255.0:3300/0,v1:10.74.255.0:6789/0],host02=[v2:10.74.249.253:3300/0,v1:10.74.249.253:6789/0],host03=[v2:10.74.251.164:3300/0,v1:10.74.251.164:6789/0]}, election epoch 6688, leader 1 host03, quorum 1,2 host03,host02

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To display the Ceph Monitor map, execute the following:

   Example

   [ceph: root@host01 /]# ceph mon stat

   or

   Example

   [ceph: root@host01 /]# ceph mon dump

3. To check the quorum status for the storage cluster, execute the following:

   [ceph: root@host01 /]# ceph quorum_status -f json-pretty

   Ceph returns the quorum status.

   Example

   {
       "election_epoch": 6686,
       "quorum": [
           0,
           1,
           2
       ],
       "quorum_names": [
           "host01",
           "host03",
           "host02"
       ],
       "quorum_leader_name": "host01",
       "quorum_age": 424884,
       "features": {
           "quorum_con": "4540138297136906239",
           "quorum_mon": [
               "kraken",
               "luminous",
               "mimic",
               "osdmap-prune",
               "nautilus",
               "octopus",
               "pacific",
               "elector-pinging"
           ]
       },
       "monmap": {
           "epoch": 3,
           "fsid": "499829b4-832f-11eb-8d6d-001a4a000635",
           "modified": "2021-03-15T04:51:38.621737Z",
           "created": "2021-03-12T12:35:16.911339Z",
           "min_mon_release": 16,
           "min_mon_release_name": "pacific",
           "election_strategy": 1,
           "disallowed_leaders: ": "",
           "stretch_mode": false,
           "features": {
               "persistent": [
                   "kraken",
                   "luminous",
                   "mimic",
                   "osdmap-prune",
                   "nautilus",
                   "octopus",
                   "pacific",
                   "elector-pinging"
               ],
               "optional": []
           },
           "mons": [
               {
                   "rank": 0,
                   "name": "host01",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.255.0:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.255.0:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.255.0:6789/0",
                   "public_addr": "10.74.255.0:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               },
               {
                   "rank": 1,
                   "name": "host03",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.251.164:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.251.164:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.251.164:6789/0",
                   "public_addr": "10.74.251.164:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               },
               {
                   "rank": 2,
                   "name": "host02",
                   "public_addrs": {
                       "addrvec": [
                           {
                               "type": "v2",
                               "addr": "10.74.249.253:3300",
                               "nonce": 0
                           },
                           {
                               "type": "v1",
                               "addr": "10.74.249.253:6789",
                               "nonce": 0
                           }
                       ]
                   },
                   "addr": "10.74.249.253:6789/0",
                   "public_addr": "10.74.249.253:6789/0",
                   "priority": 0,
                   "weight": 0,
                   "crush_location": "{}"
               }
           ]
       }
   }

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To use the socket:

   Syntax

   ceph daemon MONITOR_ID COMMAND

   Replace:

   • MONITOR_ID of the daemon

   • COMMAND with the command to run. Use help to list the available commands for a given daemon.

     To view the status of a Ceph Monitor:

     Example

     [ceph: root@host01 /]# ceph daemon mon.host01 help
     {
         "add_bootstrap_peer_hint": "add peer address as potential bootstrap peer for cluster bringup",
         "add_bootstrap_peer_hintv": "add peer address vector as potential bootstrap peer for cluster bringup",
         "compact": "cause compaction of monitor's leveldb/rocksdb storage",
         "config diff": "dump diff of current config and default config",
         "config diff get": "dump diff get <field>: dump diff of current and default config setting <field>",
         "config get": "config get <field>: get the config value",
         "config help": "get config setting schema and descriptions",
         "config set": "config set <field> <val> [<val> ...]: set a config variable",
         "config show": "dump current config settings",
         "config unset": "config unset <field>: unset a config variable",
         "connection scores dump": "show the scores used in connectivity-based elections",
         "connection scores reset": "reset the scores used in connectivity-based elections",
         "dump_historic_ops": "dump_historic_ops",
         "dump_mempools": "get mempool stats",
         "get_command_descriptions": "list available commands",
         "git_version": "get git sha1",
         "heap": "show heap usage info (available only if compiled with tcmalloc)",
         "help": "list available commands",
         "injectargs": "inject configuration arguments into running daemon",
         "log dump": "dump recent log entries to log file",
         "log flush": "flush log entries to log file",
         "log reopen": "reopen log file",
         "mon_status": "report status of monitors",
         "ops": "show the ops currently in flight",
         "perf dump": "dump perfcounters value",
         "perf histogram dump": "dump perf histogram values",
         "perf histogram schema": "dump perf histogram schema",
         "perf reset": "perf reset <name>: perf reset all or one perfcounter name",
         "perf schema": "dump perfcounters schema",
         "quorum enter": "force monitor back into quorum",
         "quorum exit": "force monitor out of the quorum",
         "sessions": "list existing sessions",
         "smart": "Query health metrics for underlying device",
         "sync_force": "force sync of and clear monitor store",
         "version": "get ceph version"
     }

     Example

     [ceph: root@host01 /]# ceph daemon mon.host01 mon_status
     
     {
         "name": "host01",
         "rank": 0,
         "state": "leader",
         "election_epoch": 120,
         "quorum": [
             0,
             1,
             2
         ],
         "quorum_age": 206358,
         "features": {
             "required_con": "2449958747317026820",
             "required_mon": [
                 "kraken",
                 "luminous",
                 "mimic",
                 "osdmap-prune",
                 "nautilus",
                 "octopus",
                 "pacific",
                 "elector-pinging"
             ],
             "quorum_con": "4540138297136906239",
             "quorum_mon": [
                 "kraken",
                 "luminous",
                 "mimic",
                 "osdmap-prune",
                 "nautilus",
                 "octopus",
                 "pacific",
                 "elector-pinging"
             ]
         },
         "outside_quorum": [],
         "extra_probe_peers": [],
         "sync_provider": [],
         "monmap": {
             "epoch": 3,
             "fsid": "81a4597a-b711-11eb-8cb8-001a4a000740",
             "modified": "2021-05-18T05:50:17.782128Z",
             "created": "2021-05-17T13:13:13.383313Z",
             "min_mon_release": 16,
             "min_mon_release_name": "pacific",
             "election_strategy": 1,
             "disallowed_leaders: ": "",
             "stretch_mode": false,
             "features": {
                 "persistent": [
                     "kraken",
                     "luminous",
                     "mimic",
                     "osdmap-prune",
                     "nautilus",
                     "octopus",
                     "pacific",
                     "elector-pinging"
                 ],
                 "optional": []
             },
             "mons": [
                 {
                     "rank": 0,
                     "name": "host01",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.41:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.41:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.41:6789/0",
                     "public_addr": "10.74.249.41:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 },
                 {
                     "rank": 1,
                     "name": "host02",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.55:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.55:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.55:6789/0",
                     "public_addr": "10.74.249.55:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 },
                 {
                     "rank": 2,
                     "name": "host03",
                     "public_addrs": {
                         "addrvec": [
                             {
                                 "type": "v2",
                                 "addr": "10.74.249.49:3300",
                                 "nonce": 0
                             },
                             {
                                 "type": "v1",
                                 "addr": "10.74.249.49:6789",
                                 "nonce": 0
                             }
                         ]
                     },
                     "addr": "10.74.249.49:6789/0",
                     "public_addr": "10.74.249.49:6789/0",
                     "priority": 0,
                     "weight": 0,
                     "crush_location": "{}"
                 }
             ]
         },
         "feature_map": {
             "mon": [
                 {
                     "features": "0x3f01cfb9fffdffff",
                     "release": "luminous",
                     "num": 1
                 }
             ],
             "osd": [
                 {
                     "features": "0x3f01cfb9fffdffff",
                     "release": "luminous",
                     "num": 3
                 }
             ]
         },
         "stretch_mode": false
     }

3. Alternatively, specify the Ceph daemon by using its socket file:

   Syntax

   ceph daemon /var/run/ceph/SOCKET_FILE COMMAND

4. To view the status of an Ceph OSD named osd.2:

   Example

   [ceph: root@host01 /]# ceph daemon /var/run/ceph/ceph-osd.2.asok status

5. To list all socket files for the Ceph processes:

   Example

   [ceph: root@host01 /]# ls /var/run/ceph

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. To retrieve a list of placement groups:

   Example

   [ceph: root@host01 /]# ceph pg dump

3. View which Ceph OSDs are in the Acting Set or in the Up Set for a given placement group:

   Syntax

   ceph pg map PG_NUM

   Example

   [ceph: root@host01 /]# ceph pg map 128

   Note

   If the Up Set and Acting Set do not match, this may be an indicator that the storage cluster rebalancing itself or of a potential problem with the storage cluster.

Procedure

1. To identify stuck placement groups, execute the following:

   Syntax

   ceph pg dump_stuck {inactive|unclean|stale|undersized|degraded [inactive|unclean|stale|undersized|degraded...]} {<int>}

   Example

   [ceph: root@host01 /]# ceph pg dump_stuck stale
   OK

Procedure

1. To find the object location, all you need is the object name and the pool name:

   Syntax

   ceph osd map POOL_NAME OBJECT_NAME

   Example

   [ceph: root@host01 /]# ceph osd map mypool myobject

Procedure

1. If you are bootstrapping your new storage cluster, you can create the service configuration .yaml file that adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services should run:

   Example

   service_type: host
   addr: host01
   hostname: host01
   location:
     root: default
     datacenter: DC1
   labels:
     - osd
     - mon
     - mgr
   ---
   service_type: host
   addr: host02
   hostname: host02
   location:
     datacenter: DC1
   labels:
     - osd
     - mon
   ---
   service_type: host
   addr: host03
   hostname: host03
   location:
     datacenter: DC1
   labels:
     - osd
     - mds
     - rgw
   ---
   service_type: host
   addr: host04
   hostname: host04
   location:
     root: default
     datacenter: DC2
   labels:
     - osd
     - mon
     - mgr
   ---
   service_type: host
   addr: host05
   hostname: host05
   location:
     datacenter: DC2
   labels:
     - osd
     - mon
   ---
   service_type: host
   addr: host06
   hostname: host06
   location:
     datacenter: DC2
   labels:
     - osd
     - mds
     - rgw
   ---
   service_type: host
   addr: host07
   hostname: host07
   labels:
     - mon
   ---
   service_type: mon
   placement:
     label: "mon"
   ---
   service_id: cephfs
   placement:
     label: "mds"
   ---
   service_type: mgr
   service_name: mgr
   placement:
     label: "mgr"
   ---
   service_type: osd
   service_id: all-available-devices
   service_name: osd.all-available-devices
   placement:
     label: "osd"
   spec:
     data_devices:
       all: true
   ---
   service_type: rgw
   service_id: objectgw
   service_name: rgw.objectgw
   placement:
     count: 2
     label: "rgw"
   spec:
     rgw_frontend_port: 8080

2. Bootstrap the storage cluster with the --apply-spec option:

   Syntax

   cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

   Example

   [root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

   Important

   You can use different command options with the cephadm bootstrap command. However, always include the --apply-spec option to use the service configuration file and configure the host locations.

Procedure

1. Add two buckets to which you plan to set the location of your non-tiebreaker monitors to the CRUSH map, specifying the bucket type as as datacenter:

   Syntax

   ceph osd crush add-bucket BUCKET_NAME BUCKET_TYPE

   Example

   [ceph: root@host01 /]# ceph osd crush add-bucket DC1 datacenter
   [ceph: root@host01 /]# ceph osd crush add-bucket DC2 datacenter

2. Move the buckets under root=default:

   Syntax

   ceph osd crush move BUCKET_NAME root=default

   Example

   [ceph: root@host01 /]# ceph osd crush move DC1 root=default
   [ceph: root@host01 /]# ceph osd crush move DC2 root=default

3. Move the OSD hosts according to the required CRUSH placement:

   Syntax

   ceph osd crush move HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph osd crush move host01 datacenter=DC1

Procedure

1. Set the location of each monitor, matching your CRUSH map:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph mon set_location host01 datacenter=DC1
   [ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC1
   [ceph: root@host01 /]# ceph mon set_location host04 datacenter=DC2
   [ceph: root@host01 /]# ceph mon set_location host05 datacenter=DC2
   [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3

2. Generate a CRUSH rule which places two copies on each data center:

   Syntax

   ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
   crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
   [ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

   1. Edit the decompiled CRUSH map file to add a new rule:

      Example

      rule stretch_rule {
              id 1 1
              type replicated
              min_size 1
              max_size 10
              step take DC1 2
              step chooseleaf firstn 2 type host
              step emit
              step take DC2 3
              step chooseleaf firstn 2 type host
              step emit
      }

      1

      The rule id has to be unique. In this example, there is only one more rule with id 0, thereby the id 1 is used, however you might need to use a different rule ID depending on the number of existing rules.

      2 3

      In this example, there are two data center buckets named DC1 and DC2.

      Note

      This rule makes the cluster have read-affinity towards data center DC1. Therefore, all the reads or writes happen through Ceph OSDs placed in DC1.

      If this is not desirable, and reads or writes are to be distributed evenly across the zones, the crush rule is the following:

      Example

      rule stretch_rule {
      id 1
      type replicated
      min_size 1
      max_size 10
      step take default
      step choose firstn 0 type datacenter
      step chooseleaf firstn 2 type host
      step emit
      }

      In this rule, the data center is selected randomly and automatically.

      See CRUSH rules for more information on firstn and indep options.

3. Inject the CRUSH map to make the rule available to the cluster:

   Syntax

   crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
   ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
   [ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

4. If you do not run the monitors in connectivity mode, set the election strategy to connectivity:

   Example

   [ceph: root@host01 /]# ceph mon set election_strategy connectivity

5. Enter stretch mode by setting the location of the tiebreaker monitor to split across the data centers:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER
   ceph mon enable_stretch_mode HOST stretch_rule datacenter

   Example

   [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3
   [ceph: root@host01 /]# ceph mon enable_stretch_mode host07 stretch_rule datacenter

   In this example the monitor mon.host07 is the tiebreaker.

   Important

   The location of the tiebreaker monitor should differ from the data centers to which you previously set the non-tiebreaker monitors. In the example above, it is data center DC3.

   Important

   Do not add this data center to the CRUSH map as it results in the following error when you try to enter stretch mode:

   Error EINVAL: there are 3 datacenters in the cluster but stretch mode currently only works with 2!

   Note

   If you are writing your own tooling for deploying Ceph, you can use a new --set-crush-location option when booting monitors, instead of running the ceph mon set_location command. This option accepts only a single bucket=location pair, for example ceph-mon --set-crush-location 'datacenter=DC1', which must match the bucket type you specified when running the enable_stretch_mode command.

6. Verify that the stretch mode is enabled successfully:

   Example

   [ceph: root@host01 /]# ceph osd dump
   
   epoch 361
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   created 2023-01-16T05:47:28.482717+0000
   modified 2023-01-17T17:36:50.066183+0000
   flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
   crush_version 31
   full_ratio 0.95
   backfillfull_ratio 0.92
   nearfull_ratio 0.85
   require_min_compat_client luminous
   min_compat_client luminous
   require_osd_release quincy
   stretch_mode_enabled true
   stretch_bucket_count 2
   degraded_stretch_mode 0
   recovering_stretch_mode 0
   stretch_mode_bucket 8

   The stretch_mode_enabled should be set to true. You can also see the number of stretch buckets, stretch mode buckets, and if the stretch mode is degraded or recovering.

7. Verify that the monitors are in an appropriate locations:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host07
   disallowed_leaders host07
   0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host07; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
   dumped monmap epoch 19

   You can also see which monitor is the tiebreaker, and the monitor election strategy.

Procedure

1. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOST_1 HOST_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls

2. Deploy the OSDs on specific hosts or on all the available devices:

   • Create an OSD from a specific device on a specific host:

     Syntax

     ceph orch daemon add osd HOST:DEVICE_PATH

     Example

     [ceph: root@host01 /]# ceph orch daemon add osd host03:/dev/sdb

   • Deploy OSDs on any available and unused devices:

     Important

     This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

3. Move the OSD hosts under the CRUSH bucket:

   Syntax

   ceph osd crush move HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph osd crush move host03 datacenter=DC1
   [ceph: root@host01 /]# ceph osd crush move host06 datacenter=DC2

   Note

   Ensure you add the same topology nodes on both sites. Issues might arise if hosts are added only on one site.

Procedure

1. To list the users in the storage cluster, execute the following:

   Example

   [ceph: root@host01 /]# ceph auth list
   installed auth entries:
   
   osd.10
   	key: AQBW7U5gqOsEExAAg/CxSwZ/gSh8iOsDV3iQOA==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   osd.11
   	key: AQBX7U5gtj/JIhAAPsLBNG+SfC2eMVEFkl3vfA==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   osd.9
   	key: AQBV7U5g1XDULhAAKo2tw6ZhH1jki5aVui2v7g==
   	caps: [mgr] allow profile osd
   	caps: [mon] allow profile osd
   	caps: [osd] allow *
   client.admin
   	key: AQADYEtgFfD3ExAAwH+C1qO7MSLE4TWRfD2g6g==
   	caps: [mds] allow *
   	caps: [mgr] allow *
   	caps: [mon] allow *
   	caps: [osd] allow *
   client.bootstrap-mds
   	key: AQAHYEtgpbkANBAANqoFlvzEXFwD8oB0w3TF4Q==
   	caps: [mon] allow profile bootstrap-mds
   client.bootstrap-mgr
   	key: AQAHYEtg3dcANBAAVQf6brq3sxTSrCrPe0pKVQ==
   	caps: [mon] allow profile bootstrap-mgr
   client.bootstrap-osd
   	key: AQAHYEtgD/QANBAATS9DuP3DbxEl86MTyKEmdw==
   	caps: [mon] allow profile bootstrap-osd
   client.bootstrap-rbd
   	key: AQAHYEtgjxEBNBAANho25V9tWNNvIKnHknW59A==
   	caps: [mon] allow profile bootstrap-rbd
   client.bootstrap-rbd-mirror
   	key: AQAHYEtgdE8BNBAAr6rLYxZci0b2hoIgH9GXYw==
   	caps: [mon] allow profile bootstrap-rbd-mirror
   client.bootstrap-rgw
   	key: AQAHYEtgwGkBNBAAuRzI4WSrnowBhZxr2XtTFg==
   	caps: [mon] allow profile bootstrap-rgw
   client.crash.host04
   	key: AQCQYEtgz8lGGhAAy5bJS8VH9fMdxuAZ3CqX5Q==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.crash.host02
   	key: AQDuYUtgqgfdOhAAsyX+Mo35M+HFpURGad7nJA==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.crash.host03
   	key: AQB98E5g5jHZAxAAklWSvmDsh2JaL5G7FvMrrA==
   	caps: [mgr] profile crash
   	caps: [mon] profile crash
   client.nfs.foo.host03
   	key: AQCgTk9gm+HvMxAAHbjG+XpdwL6prM/uMcdPdQ==
   	caps: [mon] allow r
   	caps: [osd] allow rw pool=nfs-ganesha namespace=foo
   client.nfs.foo.host03-rgw
   	key: AQCgTk9g8sJQNhAAPykcoYUuPc7IjubaFx09HQ==
   	caps: [mon] allow r
   	caps: [osd] allow rwx tag rgw *=*
   client.rgw.test_realm.test_zone.host01.hgbvnq
   	key: AQD5RE9gAQKdCRAAJzxDwD/dJObbInp9J95sXw==
   	caps: [mgr] allow rw
   	caps: [mon] allow *
   	caps: [osd] allow rwx tag rgw *=*
   client.rgw.test_realm.test_zone.host02.yqqilm
   	key: AQD0RE9gkxA4ExAAFXp3pLJWdIhsyTe2ZR6Ilw==
   	caps: [mgr] allow rw
   	caps: [mon] allow *
   	caps: [osd] allow rwx tag rgw *=*
   mgr.host01.hdhzwn
   	key: AQAEYEtg3lhIBxAAmHodoIpdvnxK0llWF80ltQ==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *
   mgr.host02.eobuuv
   	key: AQAn6U5gzUuiABAA2Fed+jPM1xwb4XDYtrQxaQ==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *
   mgr.host03.wquwpj
   	key: AQAd6U5gIzWsLBAAbOKUKZlUcAVe9kBLfajMKw==
   	caps: [mds] allow *
   	caps: [mon] profile mgr
   	caps: [osd] allow *

Procedure

1. To retrieve a specific user, key and capabilities, execute the following:

   Syntax

   ceph auth export TYPE.ID

   Example

   [ceph: root@host01 /]# ceph auth export mgr.host02.eobuuv

2. You can also use the -o FILE_NAME option.

   Syntax

   ceph auth export TYPE.ID -o FILE_NAME

   Example

   [ceph: root@host01 /]# ceph auth export osd.9 -o filename
   export auth(key=AQBV7U5g1XDULhAAKo2tw6ZhH1jki5aVui2v7g==)

Procedure

1. To add capabilities, use the form:

   Syntax

   ceph auth caps USERTYPE.USERID DAEMON 'allow [r|w|x|*|...] [pool=POOL_NAME] [namespace=NAMESPACE_NAME]'

   Example

   [ceph: root@host01 /]# ceph auth caps client.john mon 'allow r' osd 'allow rw pool=mypool'
   [ceph: root@host01 /]# ceph auth caps client.paul mon 'allow rw' osd 'allow rwx pool=mypool'
   [ceph: root@host01 /]# ceph auth caps client.brian-manager mon 'allow *' osd 'allow *'

2. To remove a capability, you may reset the capability. If you want the user to have no access to a particular daemon that was previously set, specify an empty string:

   Example

   [ceph: root@host01 /]# ceph auth caps client.ringo mon ' ' osd ' '

Procedure

1. To delete a user, use ceph auth del:

   Syntax

   ceph auth del TYPE.ID

   Example

   [ceph: root@host01 /]# ceph auth del osd.6

Procedure

1. To print a user’s authentication key to standard output, execute the following:

   Syntax

   ceph auth print-key TYPE.ID

   Example

   [ceph: root@host01 /]# ceph auth print-key osd.6
   
   AQBQ7U5gAry3JRAA3NoPrqBBThpFMcRL6Sr+5w==[ceph: root@host01 /]#

2. Printing a user’s key is useful when you need to populate client software with a user’s key, for example, libvirt.

   Syntax

   mount -t ceph HOSTNAME:/MOUNT_POINT -o name=client.user,secret=ceph auth print-key client.user

   Example

   [ceph: root@host01 /]#  mount -t ceph host02:/ceph -o name=client.user,secret=`ceph auth print-key client.user`