1.8. 使用 upmap 在 OSD 上手动重新平衡数据

流程

1. 确保 balancer 模块位于：

   示例

   [root@mon ~]# ceph mgr module ls | more
   {
       "always_on_modules": [
           "balancer",
           "crash",
           "devicehealth",
           "orchestrator_cli",
           "progress",
           "rbd_support",
           "status",
           "volumes"
       ],
       "enabled_modules": [
           "dashboard",
           "pg_autoscaler",
           "prometheus"
       ],

   Copy to Clipboard Toggle word wrap

   1. 如果 balancer 模块没有列在 always_on 或 enabled 模块中，则启用它：

      语法

      ceph mgr module enable balancer

      Copy to Clipboard Toggle word wrap

2. 将负载均衡器模式设置为 upmap ：

   语法

   ceph balancer mode upmap

   Copy to Clipboard Toggle word wrap

3. 关闭 balancer 模块：

   语法

   ceph balancer off

   Copy to Clipboard Toggle word wrap

4. 检查负载均衡器状态：

   示例

   [root@mon ~]# ceph balancer status
   {
       "plans": [],
       "active": false,
       "last_optimize_started": "",
       "last_optimize_duration": "",
       "optimize_result": "",
       "mode": "upmap"
   }

   Copy to Clipboard Toggle word wrap

5. 为 OSD 设置 norebalance 标志：

   语法

   ceph osd set norebalance

   Copy to Clipboard Toggle word wrap

6. 使用 ceph pg dump pgs_brief 命令列出存储集群中的池，各自消耗的空间。使用 grep 搜索重新映射的池。

   示例

   [root@mon ~]# ceph pg dump pgs_brief
   PG_STAT STATE                         UP               UP_PRIMARY ACTING         ACTING_PRIMARY
   dumped pgs_brief
   7.270   active+remapped+backfilling  [8,48,61]          8 [46,48,61]             46
   7.1e7   active+remapped+backfilling [73,64,74]         73 [18,64,74]             18
   7.1c1   active+remapped+backfilling  [29,14,8]         29 [29,14,24]             29
   7.17f   active+remapped+backfilling [73,71,50]         73 [50,71,69]             50
   7.16c   active+remapped+backfilling   [66,8,4]         66  [66,4,57]             66
   7.13d   active+remapped+backfilling [73,27,56]         73 [27,56,35]             27
   7.130   active+remapped+backfilling [53,47,73]         53 [53,47,72]             53
   9.e0    active+remapped+backfilling  [8,75,14]          8 [14,75,58]             14
   7.db    active+remapped+backfilling [10,57,60]         10 [10,60,50]             10
   9.7     active+remapped+backfilling [26,69,38]         26 [26,38,41]             26
   7.4a    active+remapped+backfilling [73,10,76]         73 [10,76,29]             10
   9.9a    active+remapped+backfilling [20,15,73]         20 [20,15,29]             20
   7.ac    active+remapped+backfilling   [8,74,3]          8  [3,74,37]              3
   9.c2    active+remapped+backfilling  [57,75,7]         57   [4,75,7]              4
   7.34d   active+remapped+backfilling [23,46,73]         23 [23,46,56]             23
   7.36a   active+remapped+backfilling  [40,32,8]         40 [40,32,44]             40

   Copy to Clipboard Toggle word wrap

7. 将 PG 移到您希望它们所在的 OSD。例如，将 PG 7.ac 从 OSD 8 和 3 移到 OSD 3 和 37:

   示例

   PG_STAT STATE                         UP               UP_PRIMARY ACTING         ACTING_PRIMARY
   dumped pgs_brief
   7.ac    active+remapped+backfilling   [8,74,3]          8  [3,74,37]              3
   
   [root@mon ~]# ceph osd pg-upmap-items 7.ac 8 3 3 37
   
   7.ac   active+clean                 [3,74,37]          8 [3,74,37]             3

   Copy to Clipboard Toggle word wrap

   注意

   重复此步骤，以每次移动每个重新 map 的 PG。

8. 再次使用 ceph pg dump pgs_brief 命令检查 PG 是否移至 active+clean 状态：

   示例

   [root@mon ~]# ceph pg dump pgs_brief
   PG_STAT STATE                         UP               UP_PRIMARY ACTING         ACTING_PRIMARY
   dumped pgs_brief
   7.270   active+clean                [8,48,61]          8 [46,48,61]              46
   7.1e7   active+clean                [73,64,74]         73 [18,64,74]             18
   7.1c1   active+clean                [29,14,8]          29 [29,14,24]             29
   7.17f   active+clean                [73,71,50]         73 [50,71,69]             50
   7.16c   active+clean                [66,8,4]           66  [66,4,57]             66
   7.13d   active+clean                [73,27,56]         73 [27,56,35]             27
   7.130   active+clean                [53,47,73]         53 [53,47,72]             53
   9.e0    active+clean                [8,75,14]          8 [14,75,58]              14
   7.db    active+clean                [10,57,60]         10 [10,60,50]             10
   9.7     active+clean                [26,69,38]         26 [26,38,41]             26
   7.4a    active+clean                [73,10,76]         73 [10,76,29]             10
   9.9a    active+clean                [20,15,73]         20 [20,15,29]             20
   7.ac    active+clean                [3,74,37]          8 [3,74,37]               3
   9.c2    active+clean                [57,75,7]          57 [4,75,7]                4
   7.34d   active+clean                [23,46,73]         23 [23,46,56]             23
   7.36a   active+clean                [40,32,8]          40 [40,32,44]             40

   Copy to Clipboard Toggle word wrap

   PG 移至 active+clean 所需的时间取决于 PG 和 OSD 的数量。另外，错误放入的对象数量取决于为 mgr target_max_misplaced_ratio 设置的值。为 target_max_misplaced_ratio 设置了一个更高的值，则会导致大量错误替换的对象；因此，所有 PG 都需要较长的时间才能变为 active+clean。

9. 取消设置 norebalance 标志：

   语法

   ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

10. 重新打开 balancer 模块：

    语法

    ceph balancer on

    Copy to Clipboard Toggle word wrap