Chapter 5. Generalized stretch cluster configuration for three availability zones

Procedure

1. Create the service configuration YAML file. The YAML file adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services run. The following example depends on the specific OSD and Ceph Object Gateway (RGW) configuration that is needed.

   Syntax

   service_type: host
   hostname: HOST01
   addr: IP_ADDRESS01
   labels: ['alertmanager', 'osd', 'installer', '_admin', 'mon', 'prometheus', 'mgr', 'grafana']
   location:
     root: default
     datacenter: DC1
   ---
   service_type: host
   hostname: HOST02
   addr: IP_ADDRESS02
   labels: ['osd', 'mon', 'mgr', 'rgw']
   location:
     root: default
     datacenter: DC1
   ---
   service_type: host
   hostname: HOST03
   addr: IP_ADDRESS03
   labels: ['osd', 'mon', 'mds']
   location:
     root: default
     datacenter: DC1
   ---
   service_type: host
   hostname: HOST04
   addr: IP_ADDRESS04
   labels: ['osd', '_admin', 'mon', 'mgr']
   location:
     root: default
     datacenter: DC2
   ---
   service_type: host
   hostname: HOST05
   addr: IP_ADDRESS05
   labels: ['osd', 'mon', 'mgr', 'rgw']
   location:
     root: default
     datacenter: DC2
   ---
   service_type: host
   hostname: HOST06
   addr: IP_ADDRESS06
   labels: ['osd', 'mon', 'mds']
   location:
     root: default
     datacenter: DC2
   ---
   service_type: host
   hostname: HOST07
   addr: IP_ADDRESS07
   labels: ['osd', '_admin', 'mon', 'mgr']
   location:
     root: default
     datacenter: DC3
   ---
   service_type: host
   hostname: HOST08
   addr: IP_ADDRESS08
   labels: ['osd', 'mon', 'mgr', 'rgw']
   location:
     root: default
     datacenter: DC3
   ---
   service_type: host
   hostname: HOST09
   addr: IP_ADDRESS09
   labels: ['osd', 'mon', 'mds']
   location:
     root: default
     datacenter: DC3
   ---
   service_type: mon
   service_name: mon
   placement:
     label: mon
   spec:
     crush_locations:
       HOST01:
       - datacenter=DC1
       HOST02:
       - datacenter=DC1
       HOST03:
       - datacenter=DC1
       HOST04:
       - datacenter=DC2
       HOST05:
       - datacenter=DC2
       HOST06:
       - datacenter=DC2
       HOST07:
       - datacenter=DC3
       HOST08:
       - datacenter=DC3
       HOST09:
       - datacenter=DC3
   
   ---
   service_type: mgr
   service_name: mgr
   placement:
     label: mgr
   ------
   service_type: osd
   service_id: osds
   placement:
     label: osd
   spec:
     data_devices:
        all: true
   ---------
   service_type: rgw
   service_id: rgw.rgw.1
   placement:
   label: rgw
   ------------

   For more information about changing the custom spec for OSD and Object Gateway, see the following deployment instructions: * Deploying Ceph OSDs using advanced service specifications in the Red Hat Ceph Storage Operations Guide. * Deploying the Ceph Object Gateway using the service specification in the Red Hat Ceph Storage Object Gateway Guide.

2. Bootstrap the storage cluster with the --apply-spec option.

   Syntax

   cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

   Example

   [root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

   Important

   You can use different command options with the cephadm bootstrap command but always include the --apply-spec option to use the service configuration file and configure the host locations.

3. Log into the cephadm shell.

   Syntax

   cephadm shell

   Example

   [root@host01 ~]# cephadm shell

4. Configure the public network with the subnet. For more information about configuring multiple public networks to the cluster, see Configuring multiple public networks to the cluster in the Red Hat Ceph Storage Configuration Guide.

   Syntax

   ceph config set global public_network "SUBNET_1,SUBNET_2, ..."

   Example

   [ceph: root@host01 /]# ceph config global mon public_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

5. Optional: Configure a cluster network. For more information about configuring multiple cluster networks to the cluster, see Configuring a private networkin the Red Hat Ceph Storage Configuration Guide.

   Syntax

   ceph config set global cluster_network "SUBNET_1,SUBNET_2, ..."

   Example

   [ceph: root@host01 /]# ceph config set global cluster_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

6. Optional: Verify the network configurations.

   Syntax

   ceph config dump | grep network

   Example

   [ceph: root@host01 /]# ceph config dump | grep network

7. Restart the daemons. Ceph daemons bind dynamically, so you do not have to restart the entire cluster at once if you change the network configuration for a specific daemon.

   Syntax

   ceph orch restart mon

8. Optional: To restart the cluster on the admin node as a root user, run the systemctl restart command.

   Note

   To get the FSID of the cluster, use the ceph fsid command.

   Syntax

   systemctl restart ceph-FSID_OF_CLUSTER.target

   Example

   [root@host01 ~]# systemctl restart ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad.target

1. Verify that all hosts were placed in the expected data centers, as specified in step 1 of the procedure.

   Syntax

   ceph osd tree

   Check that there are three data centers under root and that the hosts are placed in each of the expected data centers.

   Note

   The hosts with OSDs will only be present after bootstrap if OSDs are deployed during bootstrap with the specification file.

   Example

   [root@host01 ~]# ceph osd tree
   ID   CLASS  WEIGHT   TYPE NAME                                         STATUS  REWEIGHT  PRI-AFF
    -1         0.87836  root default
    -3         0.29279      datacenter DC1
    -2         0.09760          host host01-installer
     0    hdd  0.02440              osd.0                                     up   1.00000  1.00000
    12    hdd  0.02440              osd.12                                    up   1.00000  1.00000
    21    hdd  0.02440              osd.21                                    up   1.00000  1.00000
    29    hdd  0.02440              osd.29                                    up   1.00000  1.00000
    -4         0.09760          host host02
     1    hdd  0.02440              osd.1                                     up   1.00000  1.00000
     9    hdd  0.02440              osd.9                                     up   1.00000  1.00000
    18    hdd  0.02440              osd.18                                    up   1.00000  1.00000
    28    hdd  0.02440              osd.28                                    up   1.00000  1.00000
    -5         0.09760          host host03
     8    hdd  0.02440              osd.8                                     up   1.00000  1.00000
    16    hdd  0.02440              osd.16                                    up   1.00000  1.00000
    24    hdd  0.02440              osd.24                                    up   1.00000  1.00000
    34    hdd  0.02440              osd.34                                    up   1.00000  1.00000
    -7         0.29279      datacenter DC2
    -6         0.09760          host host04
     4    hdd  0.02440              osd.4                                     up   1.00000  1.00000
    13    hdd  0.02440              osd.13                                    up   1.00000  1.00000
    20    hdd  0.02440              osd.20                                    up   1.00000  1.00000
    27    hdd  0.02440              osd.27                                    up   1.00000  1.00000
    -8         0.09760          host host05
     3    hdd  0.02440              osd.3                                     up   1.00000  1.00000
    10    hdd  0.02440              osd.10                                    up   1.00000  1.00000
    19    hdd  0.02440              osd.19                                    up   1.00000  1.00000
    30    hdd  0.02440              osd.30                                    up   1.00000  1.00000
    -9         0.09760          host host06
     7    hdd  0.02440              osd.7                                     up   1.00000  1.00000
    17    hdd  0.02440              osd.17                                    up   1.00000  1.00000
    26    hdd  0.02440              osd.26                                    up   1.00000  1.00000
    35    hdd  0.02440              osd.35                                    up   1.00000  1.00000
   -11         0.29279      datacenter DC3
   -10         0.09760          host host07
     5    hdd  0.02440              osd.5                                     up   1.00000  1.00000
    14    hdd  0.02440              osd.14                                    up   1.00000  1.00000
    23    hdd  0.02440              osd.23                                    up   1.00000  1.00000
    32    hdd  0.02440              osd.32                                    up   1.00000  1.00000
   -12         0.09760          host host08
     2    hdd  0.02440              osd.2                                     up   1.00000  1.00000
    11    hdd  0.02440              osd.11                                    up   1.00000  1.00000
    22    hdd  0.02440              osd.22                                    up   1.00000  1.00000
    31    hdd  0.02440              osd.31                                    up   1.00000  1.00000
   -13         0.09760          host host09
     6    hdd  0.02440              osd.6                                     up   1.00000  1.00000
    15    hdd  0.02440              osd.15                                    up   1.00000  1.00000
    25    hdd  0.02440              osd.25                                    up   1.00000  1.00000
    33    hdd  0.02440              osd.33                                    up   1.00000  1.00000

2. From the cephadm shell, verify that the mon daemons are deployed with CRUSH locations, as specified in step 1 of the procedure.

1. Verify that the service spec and all location attributes are added correctly.

   1. Check the service name for mon daemons on the cluster, by using the ceph orch ls command.

      Example

      [root@host01 ~]# ceph orch ls
      NAME                       PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
      alertmanager               ?:9093,9094      1/1  8m ago     6d   count:1
      ceph-exporter                               9/9  8m ago     6d   *
      crash                                       9/9  8m ago     6d   *
      grafana                    ?:3000           1/1  8m ago     6d   count:1
      mds.cephfs                                  3/3  8m ago     6d   label:mds
      mgr                                         6/6  8m ago     6d   label:mgr
      mon                                         9/9  8m ago     5d   label:mon
      node-exporter              ?:9100           9/9  8m ago     6d   *
      osd.all-available-devices                    36  8m ago     6d   label:osd
      prometheus                 ?:9095           1/1  8m ago     6d   count:1
      rgw.rgw.1                  ?:80             3/3  8m ago     6d   label:rgw

   2. Confirm the mon daemon services, by using the ceph orch ls mon --export command.

      Example

      [root@host01 ~]# ceph orch ls mon --export
      service_type: mon
      service_name: mon
      placement:
        label: mon
      spec:
        crush_locations:
          host01-installer:
          - datacenter=DC1
          host02:
          - datacenter=DC1
          host03:
          - datacenter=DC1
          host04:
          - datacenter=DC2
          host05:
          - datacenter=DC2
          host06:
          - datacenter=DC2
          host07:
          - datacenter=DC3
          host08:
          - datacenter=DC3
          host09:
          - datacenter=DC3

2. Verify that the bootstrap was installed successfully, by running the cephadm shell ceph -s command. For more information, see Verifying the cluster installation.

Procedure

1. Get the most recent CRUSH map and decompile the map into a text file.

   Syntax

   ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
   crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
   [ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

2. Add the new CRUSH rule into the decompiled CRUSH map file from the previous. In this example, the rule name is 3az_rule.

   Syntax

   rule 3az_rule {
            id 1
            type replicated
            step take default
            step choose firstn 3 type datacenter
            step chooseleaf firstn 2 type host
            step emit
    }

   With this rule, the placement groups will be replicated with two copies in each of the three data centers.

3. Inject the CRUSH map to make the rule available to the cluster.

   Syntax

   crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
   ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

   Example

   [ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
   [ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

   You can verify that the rule was injected successfully, by using the following steps.

   1. List the rules on the cluster.

      Syntax

      ceph osd crush rule ls

      Example

      [ceph: root@host01 /]# ceph osd crush rule ls
      replicated_rule
      ec86_pool
      3az_rule

4. Dump the CRUSH rule.

   Syntax

   ceph osd crush rule dump CRUSH_RULE

   Example

   [ceph: root@host01 /]# ceph osd crush rule dump 3az_rule
   {
       "rule_id": 1,
       "rule_name": "3az_rule",
       "type": 1,
       "steps": [
           {
               "op": "take",
               "item": -1,
               "item_name": "default"
           },
           {
               "op": "choose_firstn",
               "num": 3,
               "type": "datacenter"
           },
           {
               "op": "chooseleaf_firstn",
               "num": 2,
               "type": "host"
           },
           {
               "op": "emit"
           }
       ]
   }

5. Set the MON election strategy to connectivity.

   Syntax

   ceph mon set election_strategy connectivity

   When updated successfully, the election_strategy is updated to 3. The default election_strategy is 1.

6. Optional: Verify the election strategy that was set in the previous step.

   Syntax

   ceph mon dump

   Check that all mon daemons are in the output and that the correct CRUSH locations are added.

   Example

   [ceph: root@host01 /]# ceph mon dump
   epoch 19
   fsid b556497a-693a-11ef-b9d1-fa163e841fd7
   last_changed 2024-09-03T12:47:08.419495+0000
   created 2024-09-02T14:50:51.490781+0000
   min_mon_release 19 (squid)
   election_strategy: 3
   0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1}
   1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1}
   2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1}
   3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2}
   4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3}
   5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2}
   6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3}
   7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3}
   8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2}
   dumped monmap epoch 19

7. Set the pool to associate with three availability zone stretch clusters. For more information about available pool values, see Pool values in the Red Hat Ceph Storage Storage Strategies Guide.

   Syntax

   ceph osd pool stretch set _POOL_NAME_ _PEERING_CRUSH_BUCKET_COUNT_ _PEERING_CRUSH_BUCKET_TARGET_ _PEERING_CRUSH_BUCKET_BARRIER_ _CRUSH_RULE_ _SIZE_ _MIN_SIZE_ [--yes-i-really-mean-it]

   Replace the variables as follows:

   POOL_NAME

   The name of the pool. It must be an existing pool, this command doesn’t create a new pool.

   PEERING_CRUSH_BUCKET_COUNT

   The value is used along with peering_crush_bucket_barrier to determined whether the set of OSDs in the chosen acting set can peer with each other, based on the number of distinct buckets there are in the acting set.

   PEERING_CRUSH_BUCKET_TARGET

   This value is used along with peering_crush_bucket_barrier and size to calculate the value bucket_max which limits the number of OSDs in the same bucket from getting chose to be in the acting set of a PG.

   PEERING_CRUSH_BUCKET_BARRIER

   The type of bucket a pool is stretched across. For example, rack, row, or datacenter.

   CRUSH_RULE

   The crush rule to use for the stretch pool. The type of pool must match the type of crush_rule (replicated or erasure).

   SIZE

   The number of replicas for objects in the stretch pool.

   MIN_SIZE

   The minimum number of replicas required for I/O in the stretch pool.

   Important

   The `--yes-i-really-mean-it flag is required when setting the PEERING_CRUSH_BUCKET_COUNT and PEERING_CRUSH_BUCKET_TARGET to be more than the number of buckets in the CRUSH map. Use the optional flag to confirm that you want to bypass the safety checks and set the values for a stretch pool.

   Example

   [ceph: root@host01 /]# ceph osd pool stretch set pool01 2 3 datacenter 3az_rule 6 3

   Note

   To revert a pool to a nonstretched cluster, use the ceph osd pool stretch unset POOL_NAME command. Using this command does not unset the crush_rule, size, and min_size values. If needed, these need to be reset manually.

   A success message is emitted that the pool stretch values were set correctly.

8. Optional: Verify the pools associated with the stretch clusters, by using the ceph osd pool stretch show commands.

   Example

   [ceph: root@host01 /]# ceph osd pool stretch show pool01
   pool: pool01
   pool_id: 1
   is_stretch_pool: 1
   peering_crush_bucket_count: 2
   peering_crush_bucket_target: 3
   peering_crush_bucket_barrier: 8
   crush_rule: 3az_rule
   size: 6
   min_size: 3

Procedure

1. From the node that contains the admin keyring, install the storage cluster’s public SSH key in the root user’s authorized_keys file on the new host.

   Syntax

   ssh-copy-id -f -i /etc/ceph/ceph.pub user@NEWHOST

   Example

   [ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host11
   [ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host12

2. Optional: Verify the status of the storage cluster and that each new host has been added by using the ceph orch host ls command. See that the new host has been added and that the Status of each host is blank in the output.

3. List the available devices to deploy OSDs.

   Deploy in one of the following ways:

   • Create an OSD from a specific device on a specific host.

     Syntax

     ceph orch daemon add osd _HOST_:_DEVICE_PATH_

     Example

     [ceph: root@host10 /]# ceph orch daemon add osd host11:/dev/sdb

   • Deploy OSDs on any available and unused devices.

     Important

     This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.

     Syntax

     ceph orch apply osd --all-available-devices

4. Move the OSD hosts under the CRUSH bucket.

   Syntax

   ceph osd crush move HOST datacenter=DATACENTER

   Example

   [ceph: root@host10 /]# ceph osd crush move host10 datacenter=DC1
   [ceph: root@host10 /]# ceph osd crush move host11 datacenter=DC2
   [ceph: root@host10 /]# ceph osd crush move host12 datacenter=DC3

   Note

   Ensure you add the same topology nodes on all sites. Issues might arise if hosts are added only on one site.