Inicio
Productos
Red Hat Ceph Storage
8
Administration Guide
Chapter 5. Generalized stretch cluster configuration for three availability zones

Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Generalized stretch cluster configuration for three availability zones

As a storage administrator, you can configure a generalized stretch cluster configuration for three availability zones with Ceph OSDs.

Ceph can withstand the loss of Ceph OSDs because of its network and cluster, which are equally reliable with failures randomly distributed across the CRUSH map. If a number of OSDs are shut down, the remaining OSDs and monitors still manage to operate.

Using a single cluster limits data availability to a single location with a single point of failure. However, in some situations, higher availability might be required. Using three availability zones allows the cluster to withstand power loss and even a full data center loss in the event of a natural disaster.

With a generalized stretch cluster configuration for three availability zones, three data centers are supported, with each site holding two copies of the data. This helps ensure that even during a data center outage, the data remains accessible and writeable from another site. With this configuration, the pool replication size is 6 and the pool min_size is 3.

Note

The standard Ceph configuration survives many failures of the network or data centers and it never compromises data consistency. If you restore enough Ceph servers following a failure, it recovers. Ceph maintains availability if you lose a data center, but can still form a quorum of monitors and have all the data available with enough copies to satisfy pools’ min_size, or CRUSH rules that replicate again to meet the size.

5.1. Generalized stretch cluster deployment limitations
Copiar enlace

When using generalized stretch clusters, the following limitations should be considered.

Generalized stretch cluster configuration for three availability zones does not support I/O operations during a netsplit scenario between two or more zones. While the cluster remains accessible for basic Ceph commands, I/O usage remains unavailable until the netsplit is resolved. This is different from stretch mode, where the tiebreaker monitor can isolate one zone of the cluster and continue I/O operations in degraded mode during a netsplit. For more information about stretch mode, see Stretch mode for a storage cluster.
In a three availability zone configuration, Red Hat Ceph Storage is designed to tolerate multiple host failures. However, if more than 25% of the OSDs in the cluster go down, Ceph may stop marking OSDs as out. This behavior is controlled by the mon_osd_min_in_ratio parameter. By default, mon_osd_min_in_ratio is set to 0.75, meaning that at least 75% of the OSDs in the cluster must remain in (active) before any additional OSDs can be marked out. This setting prevents too many OSDs from being marked out as this might lead to significant data movement. The data movement can cause high client I/O impact and long recovery times when the OSDs are returned to service.
If Red Hat Ceph Storage stops marking OSDs as out, some placement groups (PGs) may fail to rebalance to surviving OSDs, potentially leading to inactive placement groups (PGs).
Important
While adjusting the mon_osd_min_in_ratio value can allow more OSDs to be marked out and trigger rebalancing, this should be done with caution. For more information on the mon_osd_min_in_ratio parameter, see Ceph Monitor and OSD configuration options.

5.2. Generalized stretch cluster deployment requirements
Copiar enlace

This information details important hardware, software, and network requirements that are needed for deploying a generalized stretch cluster configuration for three availability zones.

5.2.1. Hardware requirements
Copiar enlace

Use the following minimum hardware requirements before deploying generalized stretch cluster configuration for three availability zones. The following table lists the physical server locations and Ceph component layout for an example three availability zone deployment.

Expand

Table 5.1. Hardware requirements
Host name	Datacenter	Ceph services
host01	DC1	OSD+MON+MGR
host02	DC1	OSD+MON+MGR+RGW
host03	DC1	OSD+MON+MDS
host04	DC2	OSD+MON+MGR
host05	DC2	OSD+MON+MGR+RGW
host06	DC2	OSD+MON+MDS
host07	DC3	OSD+MON+MGR
host08	DC3	OSD+MON+MGR+RGW
host09	DC3	OSD+MON+MDS

5.2.2. Network configuration requirements
Copiar enlace

Use the following network configuration requirements before deploying generalized stretch cluster configuration for three availability zones.

Have two separate networks, one public network and one cluster network.
Have three different data centers that support VLANS and subnets for Ceph cluster and public networks for all data centers.
Note
You can use different subnets for each of the data centers.
The latencies between data centers running the Red Hat Ceph Storage Object Storage Devices (OSDs) cannot exceed 10 ms RTT.

For more information about network considerations, see Network considerations for Red Hat Ceph Storage in the Red Hat Ceph Storage Hardware Guide.

5.2.3. Cluster setup requirements
Copiar enlace

Ensure that the hostname is configured by using the bare or short hostname in all hosts.

Syntax

hostnamectl set-hostname SHORT_NAME

hostnamectl set-hostname SHORT_NAME

Copy to Clipboard

Toggle word wrap

Note

The hostname command should only return the short hostname, when run on all nodes. If the FQDN is returned, the cluster configuration will not be successful.

5.3. Bootstrapping the Ceph cluster with a specification file
Copiar enlace

Deploy the generalized stretch cluster by setting the CRUSH location to the daemons in the cluster with the spec configuration file.

Set the CRUSH location to the daemons in the cluster with a service configuration file. Use the configuration file to add the hosts to the proper locations during deployment.

For more information about Ceph bootstrapping and different cephadm bootstrap command options, see Bootstrapping a new storage cluster in the Red Hat Ceph Storage Installation Guide.

Important

Run cephadm bootstrap on the node that you want to be the initial Monitor node in the cluster. The IP_ADDRESS option should be the IP address of the node you are using to run cephadm bootstrap.

Note

If the storage cluster includes multiple networks and interfaces, be sure to choose a network that is accessible by any node that uses the storage cluster.
To deploy a storage cluster by using IPV6 addresses, use the IPV6 address format for the --mon-ip <IP_ADDRESS> option. For example: cephadm bootstrap --mon-ip 2620:52:0:880:225:90ff:fefc:2536 --registry-json /etc/mylogin.json.
To route the internal cluster traffic over the public network, omit the --cluster-network SUBNET option.

Within this procedure the network Classless Inter-Domain Routing (CIDR) is referred to as subnet.

Prerequisites

Be sure that you have root-level access to the nodes.

Procedure

Create the service configuration YAML file. The YAML file adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services run. The following example depends on the specific OSD and Ceph Object Gateway (RGW) configuration that is needed.

Syntax

service_type: host
hostname: HOST01
addr: IP_ADDRESS01
labels: ['alertmanager', 'osd', 'installer', '_admin', 'mon', 'prometheus', 'mgr', 'grafana']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST02
addr: IP_ADDRESS02
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST03
addr: IP_ADDRESS03
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST04
addr: IP_ADDRESS04
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST05
addr: IP_ADDRESS05
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST06
addr: IP_ADDRESS06
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST07
addr: IP_ADDRESS07
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST08
addr: IP_ADDRESS08
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST09
addr: IP_ADDRESS09
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC3
---
service_type: mon
service_name: mon
placement:
  label: mon
spec:
  crush_locations:
    HOST01:
    - datacenter=DC1
    HOST02:
    - datacenter=DC1
    HOST03:
    - datacenter=DC1
    HOST04:
    - datacenter=DC2
    HOST05:
    - datacenter=DC2
    HOST06:
    - datacenter=DC2
    HOST07:
    - datacenter=DC3
    HOST08:
    - datacenter=DC3
    HOST09:
    - datacenter=DC3

---
service_type: mgr
service_name: mgr
placement:
  label: mgr
------
service_type: osd
service_id: osds
placement:
  label: osd
spec:
  data_devices:
     all: true
---------
service_type: rgw
service_id: rgw.rgw.1
placement:
label: rgw
------------

service_type: host
hostname: HOST01
addr: IP_ADDRESS01
labels: ['alertmanager', 'osd', 'installer', '_admin', 'mon', 'prometheus', 'mgr', 'grafana']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST02
addr: IP_ADDRESS02
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST03
addr: IP_ADDRESS03
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST04
addr: IP_ADDRESS04
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST05
addr: IP_ADDRESS05
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST06
addr: IP_ADDRESS06
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST07
addr: IP_ADDRESS07
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST08
addr: IP_ADDRESS08
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST09
addr: IP_ADDRESS09
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC3
---
service_type: mon
service_name: mon
placement:
  label: mon
spec:
  crush_locations:
    HOST01:
    - datacenter=DC1
    HOST02:
    - datacenter=DC1
    HOST03:
    - datacenter=DC1
    HOST04:
    - datacenter=DC2
    HOST05:
    - datacenter=DC2
    HOST06:
    - datacenter=DC2
    HOST07:
    - datacenter=DC3
    HOST08:
    - datacenter=DC3
    HOST09:
    - datacenter=DC3

---
service_type: mgr
service_name: mgr
placement:
  label: mgr
------
service_type: osd
service_id: osds
placement:
  label: osd
spec:
  data_devices:
     all: true
---------
service_type: rgw
service_id: rgw.rgw.1
placement:
label: rgw
------------

Copy to Clipboard

Toggle word wrap

For more information about changing the custom spec for OSD and Object Gateway, see the following deployment instructions: * Deploying Ceph OSDs using advanced service specifications in the Red Hat Ceph Storage Operations Guide. * Deploying the Ceph Object Gateway using the service specification in the Red Hat Ceph Storage Object Gateway Guide.

Bootstrap the storage cluster with the --apply-spec option.

Syntax

cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

Copy to Clipboard

Toggle word wrap

Example

cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

[root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

Copy to Clipboard

Toggle word wrap

Important

You can use different command options with the cephadm bootstrap command but always include the --apply-spec option to use the service configuration file and configure the host locations.

Log into the cephadm shell.
Syntax
```
cephadm shell
```
```
cephadm shell
```
Copy to Clipboard Toggle word wrap
Example
```
cephadm shell
```
```
[root@host01 ~]# cephadm shell
```
Copy to Clipboard Toggle word wrap

Configure the public network with the subnet. For more information about configuring multiple public networks to the cluster, see Configuring multiple public networks to the cluster in the Red Hat Ceph Storage Configuration Guide.

Syntax

ceph config set global public_network "SUBNET_1,SUBNET_2, ..."

ceph config set global public_network "SUBNET_1,SUBNET_2, ..."

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph config global mon public_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

[ceph: root@host01 /]# ceph config global mon public_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

Copy to Clipboard

Toggle word wrap

Optional: Configure a cluster network. For more information about configuring multiple cluster networks to the cluster, see Configuring a private networkin the Red Hat Ceph Storage Configuration Guide.

Syntax

ceph config set global cluster_network "SUBNET_1,SUBNET_2, ..."

ceph config set global cluster_network "SUBNET_1,SUBNET_2, ..."

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph config set global cluster_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

[ceph: root@host01 /]# ceph config set global cluster_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

Copy to Clipboard

Toggle word wrap

Optional: Verify the network configurations.

Syntax

ceph config dump | grep network

ceph config dump | grep network

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph config dump | grep network

[ceph: root@host01 /]# ceph config dump | grep network

Copy to Clipboard

Toggle word wrap

Restart the daemons. Ceph daemons bind dynamically, so you do not have to restart the entire cluster at once if you change the network configuration for a specific daemon.
Syntax
```
ceph orch restart mon
```
```
ceph orch restart mon
```
Copy to Clipboard Toggle word wrap
Optional: To restart the cluster on the admin node as a root user, run the systemctl restart command.
Note
To get the FSID of the cluster, use the ceph fsid command.
Syntax
```
systemctl restart ceph-FSID_OF_CLUSTER.target
```
```
systemctl restart ceph-FSID_OF_CLUSTER.target
```
Copy to Clipboard Toggle word wrap
Example
```
systemctl restart ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad.target
```
```
[root@host01 ~]# systemctl restart ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad.target
```
Copy to Clipboard Toggle word wrap

Verification

Verify the specification file details and that the bootstrap was installed successfully.

Verify that all hosts were placed in the expected data centers, as specified in step 1 of the procedure.

Syntax

ceph osd tree

ceph osd tree

Copy to Clipboard

Toggle word wrap

Check that there are three data centers under root and that the hosts are placed in each of the expected data centers.

Note

The hosts with OSDs will only be present after bootstrap if OSDs are deployed during bootstrap with the specification file.

Example

ceph osd tree

[root@host01 ~]# ceph osd tree
ID   CLASS  WEIGHT   TYPE NAME                                         STATUS  REWEIGHT  PRI-AFF
 -1         0.87836  root default
 -3         0.29279      datacenter DC1
 -2         0.09760          host host01-installer
  0    hdd  0.02440              osd.0                                     up   1.00000  1.00000
 12    hdd  0.02440              osd.12                                    up   1.00000  1.00000
 21    hdd  0.02440              osd.21                                    up   1.00000  1.00000
 29    hdd  0.02440              osd.29                                    up   1.00000  1.00000
 -4         0.09760          host host02
  1    hdd  0.02440              osd.1                                     up   1.00000  1.00000
  9    hdd  0.02440              osd.9                                     up   1.00000  1.00000
 18    hdd  0.02440              osd.18                                    up   1.00000  1.00000
 28    hdd  0.02440              osd.28                                    up   1.00000  1.00000
 -5         0.09760          host host03
  8    hdd  0.02440              osd.8                                     up   1.00000  1.00000
 16    hdd  0.02440              osd.16                                    up   1.00000  1.00000
 24    hdd  0.02440              osd.24                                    up   1.00000  1.00000
 34    hdd  0.02440              osd.34                                    up   1.00000  1.00000
 -7         0.29279      datacenter DC2
 -6         0.09760          host host04
  4    hdd  0.02440              osd.4                                     up   1.00000  1.00000
 13    hdd  0.02440              osd.13                                    up   1.00000  1.00000
 20    hdd  0.02440              osd.20                                    up   1.00000  1.00000
 27    hdd  0.02440              osd.27                                    up   1.00000  1.00000
 -8         0.09760          host host05
  3    hdd  0.02440              osd.3                                     up   1.00000  1.00000
 10    hdd  0.02440              osd.10                                    up   1.00000  1.00000
 19    hdd  0.02440              osd.19                                    up   1.00000  1.00000
 30    hdd  0.02440              osd.30                                    up   1.00000  1.00000
 -9         0.09760          host host06
  7    hdd  0.02440              osd.7                                     up   1.00000  1.00000
 17    hdd  0.02440              osd.17                                    up   1.00000  1.00000
 26    hdd  0.02440              osd.26                                    up   1.00000  1.00000
 35    hdd  0.02440              osd.35                                    up   1.00000  1.00000
-11         0.29279      datacenter DC3
-10         0.09760          host host07
  5    hdd  0.02440              osd.5                                     up   1.00000  1.00000
 14    hdd  0.02440              osd.14                                    up   1.00000  1.00000
 23    hdd  0.02440              osd.23                                    up   1.00000  1.00000
 32    hdd  0.02440              osd.32                                    up   1.00000  1.00000
-12         0.09760          host host08
  2    hdd  0.02440              osd.2                                     up   1.00000  1.00000
 11    hdd  0.02440              osd.11                                    up   1.00000  1.00000
 22    hdd  0.02440              osd.22                                    up   1.00000  1.00000
 31    hdd  0.02440              osd.31                                    up   1.00000  1.00000
-13         0.09760          host host09
  6    hdd  0.02440              osd.6                                     up   1.00000  1.00000
 15    hdd  0.02440              osd.15                                    up   1.00000  1.00000
 25    hdd  0.02440              osd.25                                    up   1.00000  1.00000
 33    hdd  0.02440              osd.33                                    up   1.00000  1.00000

Copy to Clipboard

Toggle word wrap

From the cephadm shell, verify that the mon daemons are deployed with CRUSH locations, as specified in step 1 of the procedure.

Syntax

ceph mon dump

ceph mon dump

Copy to Clipboard

Toggle word wrap

+ Check that all mon daemons are in the output and that the correct CRUSH locations are added.

+ .Example --- [root@host01 ~]# ceph mon dump epoch 19 fsid b556497a-693a-11ef-b9d1-fa163e841fd7 last_changed 2024-09-03T12:47:08.419495+0000 created 2024-09-02T14:50:51.490781+0000 min_mon_release 19 (squid) election_strategy: 3 0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1} 1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1} 2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1} 3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2} 4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3} 5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2} 6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3} 7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3} 8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2} dumped monmap epoch 19 ---

Verify that the service spec and all location attributes are added correctly.

Check the service name for mon daemons on the cluster, by using the ceph orch ls command.

Example

ceph orch ls

[root@host01 ~]# ceph orch ls
NAME                       PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
alertmanager               ?:9093,9094      1/1  8m ago     6d   count:1
ceph-exporter                               9/9  8m ago     6d   *
crash                                       9/9  8m ago     6d   *
grafana                    ?:3000           1/1  8m ago     6d   count:1
mds.cephfs                                  3/3  8m ago     6d   label:mds
mgr                                         6/6  8m ago     6d   label:mgr
mon                                         9/9  8m ago     5d   label:mon
node-exporter              ?:9100           9/9  8m ago     6d   *
osd.all-available-devices                    36  8m ago     6d   label:osd
prometheus                 ?:9095           1/1  8m ago     6d   count:1
rgw.rgw.1                  ?:80             3/3  8m ago     6d   label:rgw

Copy to Clipboard

Toggle word wrap

Confirm the mon daemon services, by using the ceph orch ls mon --export command.

Example

ceph orch ls mon --export

[root@host01 ~]# ceph orch ls mon --export
service_type: mon
service_name: mon
placement:
  label: mon
spec:
  crush_locations:
    host01-installer:
    - datacenter=DC1
    host02:
    - datacenter=DC1
    host03:
    - datacenter=DC1
    host04:
    - datacenter=DC2
    host05:
    - datacenter=DC2
    host06:
    - datacenter=DC2
    host07:
    - datacenter=DC3
    host08:
    - datacenter=DC3
    host09:
    - datacenter=DC3

Copy to Clipboard

Toggle word wrap

Verify that the bootstrap was installed successfully, by running the cephadm shell ceph -s command. For more information, see Verifying the cluster installation.

5.4. Enabling three availability zones on the pool
Copiar enlace

Use this information to enable and integrate three availability zones within a generalized stretch cluster configuration.

Prerequisites

Before you begin, make sure that you have the following prerequisites in place: * Root-level access to the nodes. * The CRUSH location is set to the hosts.

Procedure

Get the most recent CRUSH map and decompile the map into a text file.

Syntax

ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
[ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

[ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
[ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

Copy to Clipboard

Toggle word wrap

Add the new CRUSH rule into the decompiled CRUSH map file from the previous. In this example, the rule name is 3az_rule.

Syntax

rule 3az_rule {
         id 1
         type replicated
         step take default
         step choose firstn 3 type datacenter
         step chooseleaf firstn 2 type host
         step emit
 }

rule 3az_rule {
         id 1
         type replicated
         step take default
         step choose firstn 3 type datacenter
         step chooseleaf firstn 2 type host
         step emit
 }

Copy to Clipboard

Toggle word wrap

With this rule, the placement groups will be replicated with two copies in each of the three data centers.

Inject the CRUSH map to make the rule available to the cluster.

Syntax

crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
[ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

[ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
[ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

Copy to Clipboard

Toggle word wrap

You can verify that the rule was injected successfully, by using the following steps.

List the rules on the cluster.

Syntax

ceph osd crush rule ls

ceph osd crush rule ls

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph osd crush rule ls
replicated_rule
ec86_pool
3az_rule

[ceph: root@host01 /]# ceph osd crush rule ls
replicated_rule
ec86_pool
3az_rule

Copy to Clipboard

Toggle word wrap

Dump the CRUSH rule.

Syntax

ceph osd crush rule dump CRUSH_RULE

ceph osd crush rule dump CRUSH_RULE

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host01 /]# ceph osd crush rule dump 3az_rule
{
    "rule_id": 1,
    "rule_name": "3az_rule",
    "type": 1,
    "steps": [
        {
            "op": "take",
            "item": -1,
            "item_name": "default"
        },
        {
            "op": "choose_firstn",
            "num": 3,
            "type": "datacenter"
        },
        {
            "op": "chooseleaf_firstn",
            "num": 2,
            "type": "host"
        },
        {
            "op": "emit"
        }
    ]
}

[ceph: root@host01 /]# ceph osd crush rule dump 3az_rule
{
    "rule_id": 1,
    "rule_name": "3az_rule",
    "type": 1,
    "steps": [
        {
            "op": "take",
            "item": -1,
            "item_name": "default"
        },
        {
            "op": "choose_firstn",
            "num": 3,
            "type": "datacenter"
        },
        {
            "op": "chooseleaf_firstn",
            "num": 2,
            "type": "host"
        },
        {
            "op": "emit"
        }
    ]
}

Copy to Clipboard

Toggle word wrap

Set the MON election strategy to connectivity.
Syntax
```
ceph mon set election_strategy connectivity
```
```
ceph mon set election_strategy connectivity
```
Copy to Clipboard Toggle word wrap
When updated successfully, the election_strategy is updated to 3. The default election_strategy is 1.

Optional: Verify the election strategy that was set in the previous step.

Syntax

ceph mon dump

ceph mon dump

Copy to Clipboard

Toggle word wrap

Check that all mon daemons are in the output and that the correct CRUSH locations are added.

Example

[ceph: root@host01 /]# ceph mon dump
epoch 19
fsid b556497a-693a-11ef-b9d1-fa163e841fd7
last_changed 2024-09-03T12:47:08.419495+0000
created 2024-09-02T14:50:51.490781+0000
min_mon_release 19 (squid)
election_strategy: 3
0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1}
1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1}
2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1}
3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2}
4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3}
5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2}
6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3}
7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3}
8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2}
dumped monmap epoch 19

[ceph: root@host01 /]# ceph mon dump
epoch 19
fsid b556497a-693a-11ef-b9d1-fa163e841fd7
last_changed 2024-09-03T12:47:08.419495+0000
created 2024-09-02T14:50:51.490781+0000
min_mon_release 19 (squid)
election_strategy: 3
0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1}
1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1}
2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1}
3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2}
4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3}
5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2}
6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3}
7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3}
8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2}
dumped monmap epoch 19

Copy to Clipboard

Toggle word wrap

Set the pool to associate with three availability zone stretch clusters. For more information about available pool values, see Pool values in the Red Hat Ceph Storage Storage Strategies Guide.
Syntax
```
ceph osd pool stretch set _POOL_NAME_ _PEERING_CRUSH_BUCKET_COUNT_ _PEERING_CRUSH_BUCKET_TARGET_ _PEERING_CRUSH_BUCKET_BARRIER_ _CRUSH_RULE_ _SIZE_ _MIN_SIZE_ [--yes-i-really-mean-it]
```
```
ceph osd pool stretch set _POOL_NAME_ _PEERING_CRUSH_BUCKET_COUNT_ _PEERING_CRUSH_BUCKET_TARGET_ _PEERING_CRUSH_BUCKET_BARRIER_ _CRUSH_RULE_ _SIZE_ _MIN_SIZE_ [--yes-i-really-mean-it]
```
Copy to Clipboard Toggle word wrap
Replace the variables as follows:
POOL_NAME
The name of the pool. It must be an existing pool, this command doesn’t create a new pool.
PEERING_CRUSH_BUCKET_COUNT
The value is used along with peering_crush_bucket_barrier to determined whether the set of OSDs in the chosen acting set can peer with each other, based on the number of distinct buckets there are in the acting set.
PEERING_CRUSH_BUCKET_TARGET
This value is used along with peering_crush_bucket_barrier and size to calculate the value bucket_max which limits the number of OSDs in the same bucket from getting chose to be in the acting set of a PG.
PEERING_CRUSH_BUCKET_BARRIER
The type of bucket a pool is stretched across. For example, rack, row, or datacenter.
CRUSH_RULE
The crush rule to use for the stretch pool. The type of pool must match the type of crush_rule (replicated or erasure).
SIZE
The number of replicas for objects in the stretch pool.
MIN_SIZE
The minimum number of replicas required for I/O in the stretch pool.
Important
The `--yes-i-really-mean-it flag is required when setting the PEERING_CRUSH_BUCKET_COUNT and PEERING_CRUSH_BUCKET_TARGET to be more than the number of buckets in the CRUSH map. Use the optional flag to confirm that you want to bypass the safety checks and set the values for a stretch pool.
Example

[ceph: root@host01 /]# ceph osd pool stretch set pool01 2 3 datacenter 3az_rule 6 3

Copy to Clipboard Toggle word wrap
Note
To revert a pool to a nonstretched cluster, use the ceph osd pool stretch unset POOL_NAME command. Using this command does not unset the crush_rule, size, and min_size values. If needed, these need to be reset manually.
A success message is emitted that the pool stretch values were set correctly.

Optional: Verify the pools associated with the stretch clusters, by using the ceph osd pool stretch show commands.

Example

[ceph: root@host01 /]# ceph osd pool stretch show pool01
pool: pool01
pool_id: 1
is_stretch_pool: 1
peering_crush_bucket_count: 2
peering_crush_bucket_target: 3
peering_crush_bucket_barrier: 8
crush_rule: 3az_rule
size: 6
min_size: 3

[ceph: root@host01 /]# ceph osd pool stretch show pool01
pool: pool01
pool_id: 1
is_stretch_pool: 1
peering_crush_bucket_count: 2
peering_crush_bucket_target: 3
peering_crush_bucket_barrier: 8
crush_rule: 3az_rule
size: 6
min_size: 3

Copy to Clipboard

Toggle word wrap

5.5. Adding OSD hosts with three availability zones
Copiar enlace

You can add Ceph OSDs with three availability zones on a generalized stretch cluster. The procedure is similar to the addition of the OSD hosts on a cluster where a generalized stretch cluster is not enabled. For more information, see Adding OSDs in the Red Hat Ceph Storage Installing Guide.

Prerequisites

Before you begin, make sure that you have the following prerequisites in place: * A running Red Hat Ceph Storage cluster. * Three availability zones enabled on a cluster. For more information, see _Enabling three availability zones on the pool. * Root-level access to the nodes.

Procedure

From the node that contains the admin keyring, install the storage cluster’s public SSH key in the root user’s authorized_keys file on the new host.

Syntax

ssh-copy-id -f -i /etc/ceph/ceph.pub user@NEWHOST

ssh-copy-id -f -i /etc/ceph/ceph.pub user@NEWHOST

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host11
[ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host12

[ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host11
[ceph: root@host10 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host12

Copy to Clipboard

Toggle word wrap

Optional: Verify the status of the storage cluster and that each new host has been added by using the ceph orch host ls command. See that the new host has been added and that the Status of each host is blank in the output.
List the available devices to deploy OSDs.
Deploy in one of the following ways:
- Create an OSD from a specific device on a specific host.
  Syntax
  ceph orch daemon add osd _HOST_:_DEVICE_PATH_
  
  Copy to Clipboard Toggle word wrap
  Example
  [ceph: root@host10 /]# ceph orch daemon add osd host11:/dev/sdb
  
  Copy to Clipboard Toggle word wrap
- Deploy OSDs on any available and unused devices.
  Important
  This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.
  Syntax
  ceph orch apply osd --all-available-devices
  
  Copy to Clipboard Toggle word wrap

Move the OSD hosts under the CRUSH bucket.

Syntax

ceph osd crush move HOST datacenter=DATACENTER

ceph osd crush move HOST datacenter=DATACENTER

Copy to Clipboard

Toggle word wrap

Example

[ceph: root@host10 /]# ceph osd crush move host10 datacenter=DC1
[ceph: root@host10 /]# ceph osd crush move host11 datacenter=DC2
[ceph: root@host10 /]# ceph osd crush move host12 datacenter=DC3

[ceph: root@host10 /]# ceph osd crush move host10 datacenter=DC1
[ceph: root@host10 /]# ceph osd crush move host11 datacenter=DC2
[ceph: root@host10 /]# ceph osd crush move host12 datacenter=DC3

Copy to Clipboard

Toggle word wrap

Note

Ensure you add the same topology nodes on all sites. Issues might arise if hosts are added only on one site.

Verification

Verify that all hosts are moved to the assigned data centers, by using the ceph osd tree command.

Volver arriba

Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Generalized stretch cluster configuration for three availability zones

5.1. Generalized stretch cluster deployment limitations
Copiar enlace

5.2. Generalized stretch cluster deployment requirements
Copiar enlace

5.2.1. Hardware requirements
Copiar enlace

5.2.2. Network configuration requirements
Copiar enlace

5.2.3. Cluster setup requirements
Copiar enlace

5.3. Bootstrapping the Ceph cluster with a specification file
Copiar enlace

5.4. Enabling three availability zones on the pool
Copiar enlace

5.5. Adding OSD hosts with three availability zones
Copiar enlace

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Hacer que el código abierto sea más inclusivo

Acerca de Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Generalized stretch cluster configuration for three availability zones

5.1. Generalized stretch cluster deployment limitationsCopiar enlaceEnlace copiado en el portapapeles!

5.2. Generalized stretch cluster deployment requirementsCopiar enlaceEnlace copiado en el portapapeles!

5.2.1. Hardware requirementsCopiar enlaceEnlace copiado en el portapapeles!

5.2.2. Network configuration requirementsCopiar enlaceEnlace copiado en el portapapeles!

5.2.3. Cluster setup requirementsCopiar enlaceEnlace copiado en el portapapeles!

5.3. Bootstrapping the Ceph cluster with a specification fileCopiar enlaceEnlace copiado en el portapapeles!

5.4. Enabling three availability zones on the poolCopiar enlaceEnlace copiado en el portapapeles!

5.5. Adding OSD hosts with three availability zonesCopiar enlaceEnlace copiado en el portapapeles!

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Hacer que el código abierto sea más inclusivo

Acerca de Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

5.1. Generalized stretch cluster deployment limitations
Copiar enlace

5.2. Generalized stretch cluster deployment requirements
Copiar enlace

5.2.1. Hardware requirements
Copiar enlace

5.2.2. Network configuration requirements
Copiar enlace

5.2.3. Cluster setup requirements
Copiar enlace

5.3. Bootstrapping the Ceph cluster with a specification file
Copiar enlace

5.4. Enabling three availability zones on the pool
Copiar enlace

5.5. Adding OSD hosts with three availability zones
Copiar enlace