9.3. 将 Red Hat Ceph Storage MDS 迁移到现有集群中的新节点

流程

1. 验证 Red Hat Ceph Storage 集群是否健康并检查 MDS 状态：

   $ sudo cephadm shell -- ceph fs ls
   name: cephfs, metadata pool: manila_metadata, data pools: [manila_data ]
   
   $ sudo cephadm shell -- ceph mds stat
   cephfs:1 {0=mds.controller-2.oebubl=up:active} 2 up:standby
   
   $ sudo cephadm shell -- ceph fs status cephfs
   
   cephfs - 0 clients
   ======
   RANK  STATE         	MDS           	ACTIVITY 	DNS	INOS   DIRS   CAPS
    0	active  mds.controller-2.oebubl  Reqs:	0 /s   696	196	173  	0
     	POOL     	TYPE 	USED  AVAIL
   manila_metadata  metadata   152M   141G
     manila_data  	data	3072M   141G
     	STANDBY MDS
   mds.controller-0.anwiwd
   mds.controller-1.cwzhog

2. 检索有关 Ceph 文件系统(CephFS) MDS 状态的更多详细信息：

   $ sudo cephadm shell -- ceph fs dump
   
   e8
   enable_multiple, ever_enabled_multiple: 1,1
   default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
   legacy client fscid: 1
   
   Filesystem 'cephfs' (1)
   fs_name cephfs
   epoch   5
   flags   12 joinable allow_snaps allow_multimds_snaps
   created 2024-01-18T19:04:01.633820+0000
   modified    	2024-01-18T19:04:05.393046+0000
   tableserver 	0
   root	0
   session_timeout 60
   session_autoclose   	300
   max_file_size   1099511627776
   required_client_features    	{}
   last_failure	0
   last_failure_osd_epoch  0
   compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
   max_mds 1
   in  	0
   up  	{0=24553}
   failed
   damaged
   stopped
   data_pools  	[7]
   metadata_pool   9
   inline_data 	disabled
   balancer
   standby_count_wanted	1
   [mds.mds.controller-2.oebubl{0:24553} state up:active seq 2 addr [v2:172.17.3.114:6800/680266012,v1:172.17.3.114:6801/680266012] compat {c=[1],r=[1],i=[7ff]}]
   
   
   Standby daemons:
   
   [mds.mds.controller-0.anwiwd{-1:14715} state up:standby seq 1 addr [v2:172.17.3.20:6802/3969145800,v1:172.17.3.20:6803/3969145800] compat {c=[1],r=[1],i=[7ff]}]
   [mds.mds.controller-1.cwzhog{-1:24566} state up:standby seq 1 addr [v2:172.17.3.43:6800/2227381308,v1:172.17.3.43:6801/2227381308] compat {c=[1],r=[1],i=[7ff]}]
   dumped fsmap epoch 8

3. 检查 OSD 块列表并清理客户端列表：

   $ sudo cephadm shell -- ceph osd blocklist ls
   $ for item in $(sudo cephadm shell -- ceph osd blocklist ls | awk '{print $1}'); do
   >     sudo cephadm shell -- ceph osd blocklist rm $item;
   > done

   注意

   当文件系统客户端不响应或行为不当时，对文件系统的访问可能会强制终止。此过程称为驱除。驱除 CephFS 客户端会阻止它进一步与 MDS 守护进程和 OSD 守护进程通信。

   通常，列入黑名单的客户端无法重新连接到服务器；您必须卸载，然后重新挂载客户端。但是，允许被驱除的客户端尝试重新连接客户端很有用。由于 CephFS 使用 RADOS OSD 块列表来控制客户端驱除，因此您可以通过从 blocklist 中删除它们来允许 CephFS 客户端重新连接。

4. 获取当前属于 Red Hat Ceph Storage 集群的主机：

   [ceph: root@controller-0 /]# ceph orch host ls
   HOST                        ADDR           LABELS          STATUS
   cephstorage-0.redhat.local  192.168.24.25  osd
   cephstorage-1.redhat.local  192.168.24.50  osd
   cephstorage-2.redhat.local  192.168.24.47  osd
   controller-0.redhat.local   192.168.24.24  _admin mgr mon
   controller-1.redhat.local   192.168.24.42  mgr _admin mon
   controller-2.redhat.local   192.168.24.37  mgr _admin mon
   6 hosts in cluster

5. 将 MDS 标签应用到目标节点：

   for item in $(sudo cephadm shell --  ceph orch host ls --format json | jq -r '.[].hostname'); do
       sudo cephadm shell -- ceph orch host label add  $item mds;
   done

6. 验证所有主机是否具有 MDS 标签：

   $ sudo cephadm shell -- ceph orch host ls
   
   HOST                    	ADDR       	   LABELS
   cephstorage-0.redhat.local  192.168.24.11  osd mds
   cephstorage-1.redhat.local  192.168.24.12  osd mds
   cephstorage-2.redhat.local  192.168.24.47  osd mds
   controller-0.redhat.local   192.168.24.35  _admin mon mgr mds
   controller-1.redhat.local   192.168.24.53  mon _admin mgr mds
   controller-2.redhat.local   192.168.24.10  mon _admin mgr mds

7. 转储当前的 MDS 规格：

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ mkdir -p ${SPEC_DIR}
   $ sudo cephadm shell -- ceph orch ls --export mds > ${SPEC_DIR}/mds

8. 编辑检索到的 spec，并将 placement.hosts 部分替换为 placement.label ：

   service_type: mds
   service_id: mds
   service_name: mds.mds
   placement:
     label: mds

9. 使用 ceph 编配器 应用新的 MDS 规格：

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ sudo cephadm shell -m ${SPEC_DIR}/mds -- ceph orch apply -i /mnt/mds
   
   Scheduling new mds deployment ...

   这会增加 MDS 守护进程的数量。

10. 检查临时添加到 CephFS 中的新备用守护进程：

    $ sudo cephadm shell -- ceph fs dump
    
    Active
    
    standby_count_wanted    1
    [mds.mds.controller-0.awzplm{0:463158} state up:active seq 307 join_fscid=1 addr [v2:172.17.3.20:6802/51565420,v1:172.17.3.20:6803/51565420] compat {c=[1],r=[1],i=[7ff]}]
    
    
    Standby daemons:
    
    [mds.mds.cephstorage-1.jkvomp{-1:463800} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/2075903648,v1:172.17.3.135:6821/2075903648] compat {c=[1],r=[1],i=[7ff]}]
    [mds.mds.controller-2.gfrqvc{-1:475945} state up:standby seq 1 addr [v2:172.17.3.114:6800/2452517189,v1:172.17.3.114:6801/2452517189] compat {c=[1],r=[1],i=[7ff]}]
    [mds.mds.cephstorage-0.fqcshx{-1:476503} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}]
    [mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}]
    [mds.mds.controller-1.tyiziq{-1:499136} state up:standby seq 1 addr [v2:172.17.3.43:6800/3615018301,v1:172.17.3.43:6801/3615018301] compat {c=[1],r=[1],i=[7ff]}]

11. 要将 MDS 迁移到目标节点，请设置管理 MDS 故障转移的 MDS 关联性：

    注意

    可以为特定文件系统选择专用 MDS 作为"主动"要配置此首选项，CephFS 为名为 mds_join_fs 的 MDS 提供了配置选项，它强制执行这个关联性。当 MDS 守护进程失败时，集群监控器首选使用 mds_join_fs 与带有失败等级的文件系统名称相等的待机守护进程。如果没有与文件系统名称相等的 mds_join_fs 的待机，它会选择非限定备用作为替换。

    $ sudo cephadm shell -- ceph config set mds.mds.cephstorage-0.fqcshx mds_join_fs cephfs

    • 将 mds.mds.cephstorage-0.fqcshx 替换为上一步中检索的 cephstorage-0 上部署的守护进程。

12. 从 Controller 节点中删除标签，并强制 MDS 故障切换到目标节点：

    $ for i in 0 1 2; do sudo cephadm shell -- ceph orch host label rm "controller-$i.redhat.local" mds; done
    
    Removed label mds from host controller-0.redhat.local
    Removed label mds from host controller-1.redhat.local
    Removed label mds from host controller-2.redhat.local

    切换到目标节点在后台发生。新的活跃 MDS 是您使用 mds_join_fs 命令设置的 MDS。

13. 检查故障转移和新部署的守护进程的结果：

    $ sudo cephadm shell -- ceph fs dump
    …
    …
    standby_count_wanted    1
    [mds.mds.cephstorage-0.fqcshx{0:476503} state up:active seq 168 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}]
    
    
    Standby daemons:
    
    [mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}]
    [mds.mds.cephstorage-1.jkvomp{-1:499760} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/452139733,v1:172.17.3.135:6821/452139733] compat {c=[1],r=[1],i=[7ff]}]
    
    
    $ sudo cephadm shell -- ceph orch ls
    
    NAME                     PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
    crash                                6/6  10m ago    10d  *
    mds.mds                          3/3  10m ago    32m  label:mds
    
    
    $ sudo cephadm shell -- ceph orch ps | grep mds
    
    
    mds.mds.cephstorage-0.fqcshx  cephstorage-0.redhat.local                     running (79m)     3m ago  79m    27.2M        -  17.2.6-100.el9cp  1af7b794f353  2a2dc5ba6d57
    mds.mds.cephstorage-1.jkvomp  cephstorage-1.redhat.local                     running (79m)     3m ago  79m    21.5M        -  17.2.6-100.el9cp  1af7b794f353  7198b87104c8
    mds.mds.cephstorage-2.gnfhfe  cephstorage-2.redhat.local                     running (79m)     3m ago  79m    24.2M        -  17.2.6-100.el9cp  1af7b794f353  f3cb859e2a15