8.2. 删除 Red Hat Ceph Storage 服务

流程

1. 使用 SSH 连接到您要删除的 DistributedComputeHCI 节点，并以 root 用户身份登录。

   $ ssh heat-admin@<dcn-computehci-node>
   $ sudo su -
   #

2. 识别与您要删除的 DistributedComputeHCI 节点关联的 OSD：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dnc2-computehci2-1 ceph osd tree -c /etc/ceph/dcn2.conf
   …
   -3       0.24399     host dcn2-computehci2-1
    1   hdd 0.04880         osd.1                           up  1.00000 1.00000
    7   hdd 0.04880         osd.7                           up  1.00000 1.00000
   11   hdd 0.04880         osd.11                          up  1.00000 1.00000
   15   hdd 0.04880         osd.15                          up  1.00000 1.00000
   18   hdd 0.04880         osd.18                          up  1.00000 1.00000
   …

3. 禁用相关 Ceph 节点上的 OSD：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd out 1 7 11 15 18 -c /etc/ceph/dcn2.conf
   marked out osd.1. marked out osd.7. marked out osd.11. marked out osd.15. marked out osd.18.

4. 等待 Ceph osd 重新平衡完成。使用以下命令监控进度：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph -w -c /etc/ceph/dcn2.conf
   …
   mon.dcn2-computehci2-2 has auth_allow_insecure_global_id_reclaim set to true

   当您看到 auth_allow_insecure_global_id_reclaim 设置为 true 时，重新平衡已完成。

5. 停止并禁用 OSD：

   [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@1
   [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@7
   [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@11
   [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@15
   [root@dcn2-computehci2-1 ~]# systemctl stop ceph-osd@18
   [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@1
   Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@1.service.
   [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@7
   Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@7.service.
   [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@11
   Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@11.service.
   [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@15
   Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@15.service.
   [root@dcn2-computehci2-1 ~]# systemctl disable ceph-osd@18
   Removed /etc/systemd/system/multi-user.target.wants/ceph-osd@18.service.

6. 从 CRUSH map 中删除 OSD：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.1 -c /etc/ceph/dcn2.conf
   removed item id 1 name 'osd.1' from crush map
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.7 -c /etc/ceph/dcn2.conf
   removed item id 7 name 'osd.7' from crush map
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.11 -c /etc/ceph/dcn2.conf
   removed item id 11 name 'osd.11' from crush map
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.15 -c /etc/ceph/dcn2.conf
   removed item id 15 name 'osd.15' from crush map
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush remove osd.18 -c /etc/ceph/dcn2.conf
   removed item id 18 name 'osd.18' from crush map

7. 删除 OSD 身份验证密钥：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.1 -c /etc/ceph/dcn2.conf
   updated
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.7 -c /etc/ceph/dcn2.conf
   updated
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.11 -c /etc/ceph/dcn2.conf
   updated
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.15 -c /etc/ceph/dcn2.conf
   updated
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph auth del osd.18 -c /etc/ceph/dcn2.conf
   updated

8. 从集群中移除 OSD：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 1 -c /etc/ceph/dcn2.conf
   removed osd.1
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 7 -c /etc/ceph/dcn2.conf
   removed osd.7
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 11 -c /etc/ceph/dcn2.conf
   removed osd.11
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 15 -c /etc/ceph/dcn2.conf
   removed osd.15
   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd rm 18 -c /etc/ceph/dcn2.conf
   removed osd.18

9. 从 CRUSH map 中删除 DistributedComputeHCI 节点：

   [root@dcn2-computehci2-1 ~]# podman exec ceph-mon-dcn2-computehci2-1 ceph osd crush rm dcn2-computehci2-1 -c /etc/ceph/dcn2.conf
   removed item id -3 name 'dcn2-computehci2-1' from crush map

10. 停止并禁用 mon 服务：

    [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph
      ceph-crash@dcn2-computehci2-1.service     loaded active     running       Ceph crash dump collector
      ceph-mgr@dcn2-computehci2-1.service       loaded active     running       Ceph Manager
      ceph-mon@dcn2-computehci2-1.service       loaded active     running       Ceph Monitor
    
    [root@dcn2-computehci2-1 ~]# systemctl stop ceph-mon@dcn2-computehci2-1
    
    [root@dcn2-computehci2-1 ~]# systemctl disable ceph-mon@dcn2-computehci2-1
    Removed /etc/systemd/system/multi-user.target.wants/ceph-mon@dcn2-computehci2-1.service.

11. 使用 SSH 连接到同一集群中的另一节点，并从集群中移除该监控器。请注意输出中的 v1 和 v2 条目：

    [root@dcn2-computehci2-0 ~]# podman exec ceph-mon-dcn2-computehci2-0 ceph mon remove dcn2-computehci2-1 -c /etc/ceph/dcn2.conf
    removing mon.dcn2-computehci2-1 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors

12. 在所有 dcn2 节点上，删除上一步中输出的 /etc/ceph/dcn2.conf 中的 v1 和 v2 监控条目，并从 'mon 初始成员' 中删除节点名称：

    之前

    mon host = [v2:172.23.3.150:3300,v1:172.23.3.150:6789],*[v2:172.23.3.153:3300,v1:172.23.3.153:6789]*,[v2:172.23.3.124:3300,v1:172.23.3.124:6789] + mon initial members = dcn2-computehci2-0,*dcn2-computehci2-1*,dcn2-computehci2-2

    After

    mon host = [v2:172.23.3.150:3300,v1:172.23.3.150:6789],[v2:172.23.3.124:3300,v1:172.23.3.124:6789] + mon initial members = dcn2-computehci2-0,dcn2-computehci2-2

13. 停止并禁用 mgr 服务：

    [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph
      ceph-crash@dcn2-computehci2-1.service    loaded active     running       Ceph crash dump collector
      ceph-mgr@dcn2-computehci2-1.service      loaded active     running       Ceph Manager
    [root@dcn2-computehci2-1 ~]# systemctl stop ceph-mgr@dcn2-computehci2-1
    [root@dcn2-computehci2-1 ~]# systemctl --type=service | grep ceph
      ceph-crash@dcn2-computehci2-1.service  loaded active running Ceph crash dump collector
    [root@dcn2-computehci2-1 ~]# systemctl disable ceph-mgr@dcn2-computehci2-1
    Removed /etc/systemd/system/multi-user.target.wants/ceph-mgr@dcn2-computehci2-1.service.

14. 验证节点的 mgr 服务是否已从集群中移除。

    [root@dcn2-computehci2-0 ~]# podman exec ceph-mon-dcn2-computehci2-0 ceph -s -c /etc/ceph/dcn2.conf
    cluster:
        id:     b9b53581-d590-41ac-8463-2f50aa985001
        health: HEALTH_WARN
                3 pools have too many placement groups
                mons are allowing insecure global_id reclaim
    
      services:
        mon: 2 daemons, quorum dcn2-computehci2-2,dcn2-computehci2-0 (age 2h)
        mgr: dcn2-computehci2-2(active, since 20h), standbys: dcn2-computehci2-0 

    1

    
        osd: 15 osds: 15 up (since 3h), 15 in (since 3h)
    
      data:
        pools:   3 pools, 384 pgs
        objects: 32 objects, 88 MiB
        usage:   16 GiB used, 734 GiB / 750 GiB avail
        pgs:     384 active+clean

    1

    当 mgr 服务被成功移除时，将不再列出 mgr 服务的节点。