Chapter 4. Stretch clusters for Ceph storage

As a storage administrator, you can configure stretch clusters by entering stretch mode with 2-site clusters.

Red Hat Ceph Storage is capable of withstanding the loss of Ceph OSDs because of its network and cluster, which are equally reliable with failures randomly distributed across the CRUSH map. If a number of OSDs is shut down, the remaining OSDs and monitors still manage to operate.

However, this might not be the best solution for some stretched cluster configurations where a significant part of the Ceph cluster can use only a single network component. The example is a single cluster located in multiple data centers, for which the user wants to sustain a loss of a full data center.

The standard configuration is with two data centers. Other configurations are in clouds or availability zones. Each site holds two copies of the data, therefore, the replication size is four. The third site should have a tiebreaker monitor, this can be a virtual machine or high-latency compared to the main sites. This monitor chooses one of the sites to restore data if the network connection fails and both data centers remain active.

Note

The standard Ceph configuration survives many failures of the network or data centers and it never compromises data consistency. If you restore enough Ceph servers following a failure, it recovers. Ceph maintains availability if you lose a data center, but can still form a quorum of monitors and have all the data available with enough copies to satisfy pools’ min_size, or CRUSH rules that replicate again to meet the size.

Note

There are no additional steps to power down a stretch cluster. You can see the Powering down and rebooting Red Hat Ceph Storage cluster for more information.

Stretch cluster failures

Red Hat Ceph Storage never compromises on data integrity and consistency. If there is a network failure or a loss of nodes and the services can still be restored, Ceph returns to normal functionality on its own.

However, there are situations where you lose data availability even if you have enough servers available to meet Ceph’s consistency and sizing constraints, or where you unexpectedly do not meet the constraints.

First important type of failure is caused by inconsistent networks. If there is a network split, Ceph might be unable to mark OSD as down to remove it from the acting placement group (PG) sets despite the primary OSD being unable to replicate data. When this happens, the I/O is not permitted because Ceph cannot meet its durability guarantees.

The second important category of failures is when it appears that you have data replicated across data enters, but the constraints are not sufficient to guarantee this. For example, you might have data centers A and B, and the CRUSH rule targets three copies and places a copy in each data center with a min_size of 2. The PG might go active with two copies in site A and no copies in site B, which means that if you lose site A, you lose the data and Ceph cannot operate on it. This situation is difficult to avoid with standard CRUSH rules.

4.1. Stretch mode for a storage cluster

To configure stretch clusters, you must enter the stretch mode. When stretch mode is enabled, the Ceph OSDs only take PGs as active when they peer across data centers, or whichever other CRUSH bucket type you specified, assuming both are active. Pools increase in size from the default three to four, with two copies on each site.

In stretch mode, Ceph OSDs are only allowed to connect to monitors within the same data center. New monitors are not allowed to join the cluster without specified location.

If all the OSDs and monitors from a data center become inaccessible at once, the surviving data center will enter a degraded stretch mode. This issues a warning, reduces the min_size to 1, and allows the cluster to reach an active state with the data from the remaining site.

Note

The degraded state also triggers warnings that the pools are too small, because the pool size does not get changed. However, a special stretch mode flag prevents the OSDs from creating extra copies in the remaining data center, therefore it still keeps 2 copies.

When the missing data center becomes accesible again, the cluster enters recovery stretch mode. This changes the warning and allows peering, but still requires only the OSDs from the data center, which was up the whole time.

When all PGs are in a known state and are not degraded or incomplete, the cluster goes back to the regular stretch mode, ends the warning, and restores min_size to its starting value 2. The cluster again requires both sites to peer, not only the site that stayed up the whole time, therefore you can fail over to the other site, if necessary.

Stretch mode limitations

It is not possible to exit from stretch mode once it is entered.
You cannot use erasure-coded pools with clusters in stretch mode. You can neither enter the stretch mode with erasure-coded pools, nor create an erasure-coded pool when the stretch mode is active.
Stretch mode with no more than two sites is supported.
The weights of the two sites should be the same. If they are not, you receive the following error:
Example
```
[ceph: root@host01 /]# ceph mon enable_stretch_mode host05 stretch_rule datacenter

Error EINVAL: the 2 datacenter instances in the cluster have differing weights 25947 and 15728 but stretch mode currently requires they be the same!
```
To achieve same weights on both sites, the Ceph OSDs deployed in the two sites should be of equal size, that is, storage capacity in the first site is equivalent to storage capacity in the second site.
While it is not enforced, you should run two Ceph monitors on each site and a tiebreaker, for a total of five. This is because OSDs can only connect to monitors in their own site when in stretch mode.
You have to create your own CRUSH rule, which provides two copies on each site, which totals to four on both sites.
You cannot enable stretch mode if you have existing pools with non-default size or min_size.
Because the cluster runs with min_size 1 when degraded, you should only use stretch mode with all-flash OSDs. This minimizes the time needed to recover once connectivity is restored, and minimizes the potential for data loss.

Additional Resources

See Troubleshooting clusters in stretch mode for troubleshooting steps.

4.1.1. Setting the CRUSH location for the daemons

Before you enter the stretch mode, you need to prepare the cluster by setting the CRUSH location to the daemons in the Red Hat Ceph Storage cluster. There are two ways to do this:

Bootstrap the cluster through a service configuration file, where the locations are added to the hosts as part of deployment.
Set the locations manually through ceph osd crush add-bucket and ceph osd crush move commands after the cluster is deployed.

Method 1: Bootstrapping the cluster

Prerequisites

Root-level access to the nodes.

Procedure

If you are bootstrapping your new storage cluster, you can create the service configuration .yaml file that adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services should run:

Example

service_type: host
addr: host01
hostname: host01
location:
  root: default
  datacenter: DC1
labels:
  - osd
  - mon
  - mgr
---
service_type: host
addr: host02
hostname: host02
location:
  datacenter: DC1
labels:
  - osd
  - mon
---
service_type: host
addr: host03
hostname: host03
location:
  datacenter: DC1
labels:
  - osd
  - mds
  - rgw
---
service_type: host
addr: host04
hostname: host04
location:
  root: default
  datacenter: DC2
labels:
  - osd
  - mon
  - mgr
---
service_type: host
addr: host05
hostname: host05
location:
  datacenter: DC2
labels:
  - osd
  - mon
---
service_type: host
addr: host06
hostname: host06
location:
  datacenter: DC2
labels:
  - osd
  - mds
  - rgw
---
service_type: host
addr: host07
hostname: host07
labels:
  - mon
---
service_type: mon
placement:
  label: "mon"
---
service_id: cephfs
placement:
  label: "mds"
---
service_type: mgr
service_name: mgr
placement:
  label: "mgr"
---
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
placement:
  label: "osd"
spec:
  data_devices:
    all: true
---
service_type: rgw
service_id: objectgw
service_name: rgw.objectgw
placement:
  count: 2
  label: "rgw"
spec:
  rgw_frontend_port: 8080

Bootstrap the storage cluster with the --apply-spec option:

Syntax

cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

Example

[root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

Important

You can use different command options with the cephadm bootstrap command. However, always include the --apply-spec option to use the service configuration file and configure the host locations.

Additional Resources

See Bootstrapping a new storage cluster for more information about Ceph bootstrapping and different cephadm bootstrap command options.

Method 2: Setting the locations after the deployment

Prerequisites

Root-level access to the nodes.

Procedure

Add two buckets to which you plan to set the location of your non-tiebreaker monitors to the CRUSH map, specifying the bucket type as as datacenter:

Syntax

ceph osd crush add-bucket BUCKET_NAME BUCKET_TYPE

Example

[ceph: root@host01 /]# ceph osd crush add-bucket DC1 datacenter
[ceph: root@host01 /]# ceph osd crush add-bucket DC2 datacenter

Move the buckets under root=default:

Syntax

ceph osd crush move BUCKET_NAME root=default

Example

[ceph: root@host01 /]# ceph osd crush move DC1 root=default
[ceph: root@host01 /]# ceph osd crush move DC2 root=default

Move the OSD hosts according to the required CRUSH placement:

Syntax

ceph osd crush move HOST datacenter=DATACENTER

Example

[ceph: root@host01 /]# ceph osd crush move host01 datacenter=DC1

4.1.2. Entering the stretch mode

The new stretch mode is designed to handle two sites. There is a lower risk of component availability outages with 2-site clusters.

Prerequisites

Root-level access to the nodes.
The CRUSH location is set to the hosts.

Procedure

Set the location of each monitor, matching your CRUSH map:

Syntax

ceph mon set_location HOST datacenter=DATACENTER

Example

[ceph: root@host01 /]# ceph mon set_location host01 datacenter=DC1
[ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC1
[ceph: root@host01 /]# ceph mon set_location host04 datacenter=DC2
[ceph: root@host01 /]# ceph mon set_location host05 datacenter=DC2
[ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3

Generate a CRUSH rule which places two copies on each data center:
Syntax
```
ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME
```
Example
```
[ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
[ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt
```
1. Edit the decompiled CRUSH map file to add a new rule:
  Example
```
rule stretch_rule {
        id 1 1
        type replicated
        min_size 1
        max_size 10
        step take DC1 2
        step chooseleaf firstn 2 type host
        step emit
        step take DC2 3
        step chooseleaf firstn 2 type host
        step emit
}
```
  1
  The rule id has to be unique. In this example, there is only one more rule with id 0, thereby the id 1 is used, however you might need to use a different rule ID depending on the number of existing rules.
  2 3
  In this example, there are two data center buckets named DC1 and DC2.
  Note
  This rule makes the cluster have read-affinity towards data center DC1. Therefore, all the reads or writes happen through Ceph OSDs placed in DC1.
  If this is not desirable, and reads or writes are to be distributed evenly across the zones, the CRUSH rule is the following:
  Example
  
  rule stretch_rule { id 1 type replicated min_size 1 max_size 10 step take default step choose firstn 0 type datacenter step chooseleaf firstn 2 type host step emit }
  
  In this rule, the data center is selected randomly and automatically.
  See CRUSH rules for more information on firstn and indep options.

Inject the CRUSH map to make the rule available to the cluster:

Syntax

crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

Example

[ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
[ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

If you do not run the monitors in connectivity mode, set the election strategy to connectivity:
Example
```
[ceph: root@host01 /]# ceph mon set election_strategy connectivity
```
Enter stretch mode by setting the location of the tiebreaker monitor to split across the data centers:
Syntax
```
ceph mon set_location HOST datacenter=DATACENTER
ceph mon enable_stretch_mode HOST stretch_rule datacenter
```
Example
```
[ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3
[ceph: root@host01 /]# ceph mon enable_stretch_mode host07 stretch_rule datacenter
```
In this example the monitor mon.host07 is the tiebreaker.
Important
The location of the tiebreaker monitor should differ from the data centers to which you previously set the non-tiebreaker monitors. In the example above, it is data center DC3.
Important
Do not add this data center to the CRUSH map as it results in the following error when you try to enter stretch mode:
```
Error EINVAL: there are 3 datacenters in the cluster but stretch mode currently only works with 2!
```
Note
If you are writing your own tooling for deploying Ceph, you can use a new --set-crush-location option when booting monitors, instead of running the ceph mon set_location command. This option accepts only a single bucket=location pair, for example ceph-mon --set-crush-location 'datacenter=DC1', which must match the bucket type you specified when running the enable_stretch_mode command.

Verify that the stretch mode is enabled successfully:

Example

[ceph: root@host01 /]# ceph osd dump

epoch 361
fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
created 2023-01-16T05:47:28.482717+0000
modified 2023-01-17T17:36:50.066183+0000
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 31
full_ratio 0.95
backfillfull_ratio 0.92
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client luminous
require_osd_release quincy
stretch_mode_enabled true
stretch_bucket_count 2
degraded_stretch_mode 0
recovering_stretch_mode 0
stretch_mode_bucket 8

The stretch_mode_enabled should be set to true. You can also see the number of stretch buckets, stretch mode buckets, and if the stretch mode is degraded or recovering.

Verify that the monitors are in an appropriate locations:

Example

[ceph: root@host01 /]# ceph mon dump

epoch 19
fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
last_changed 2023-01-17T04:12:05.709475+0000
created 2023-01-16T05:47:25.631684+0000
min_mon_release 16 (pacific)
election_strategy: 3
stretch_mode_enabled 1
tiebreaker_mon host07
disallowed_leaders host07
0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host07; crush_location {datacenter=DC3}
1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
dumped monmap epoch 19

You can also see which monitor is the tiebreaker, and the monitor election strategy.

Additional Resources

See Configuring monitor election strategy for more information about monitor election strategy.

4.1.3. Adding OSD hosts in stretch mode

You can add Ceph OSDs in the stretch mode. The procedure is similar to the addition of the OSD hosts on a cluster where stretch mode is not enabled.

Prerequisites

A running Red Hat Ceph Storage cluster.
Stretch mode in enabled on a cluster.
Root-level access to the nodes.

Procedure

List the available devices to deploy OSDs:

Syntax

ceph orch device ls [--hostname=HOST_1 HOST_2] [--wide] [--refresh]

Example

[ceph: root@host01 /]# ceph orch device ls

Deploy the OSDs on specific hosts or on all the available devices:
- Create an OSD from a specific device on a specific host:
  Syntax
```
ceph orch daemon add osd HOST:DEVICE_PATH
```
  Example
```
[ceph: root@host01 /]# ceph orch daemon add osd host03:/dev/sdb
```
- Deploy OSDs on any available and unused devices:
  Important
  This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.
  Example
```
[ceph: root@host01 /]# ceph orch apply osd --all-available-devices
```

Move the OSD hosts under the CRUSH bucket:

Syntax

ceph osd crush move HOST datacenter=DATACENTER

Example

[ceph: root@host01 /]# ceph osd crush move host03 datacenter=DC1
[ceph: root@host01 /]# ceph osd crush move host06 datacenter=DC2

Note

Ensure you add the same topology nodes on both sites. Issues might arise if hosts are added only on one site.

Additional Resources

See Adding OSDs for more information about the addition of Ceph OSDs.

4.2. Read affinity in stretch clusters

Read Affinity reduces cross-zone traffic by keeping the data access within the respective data centers.

For stretched clusters deployed in multi-zone environments, the read affinity topology implementation provides a mechanism to help keep traffic within the data center it originated from. Ceph Object Gateway volumes have the ability to read data from an OSD in proximity to the client, according to OSD locations defined in the CRUSH map and topology labels on nodes.

For example, a stretch cluster contains a Ceph Object Gateway primary OSD and replicated OSDs spread across two data centers A and B. If a GET action is performed on an Object in data center A, the READ operation is performed on the data of the OSDs closest to the client in data center A.

4.2.1. Performing localized reads

You can perform a localized read on a replicated pool in a stretch cluster. When a localized read request is made on a replicated pool, Ceph selects the local OSDs closest to the client based on the client location specified in crush_location.

Prerequisites

A stretch cluster with two data centers and Ceph Object Gateway configured on both.
A user created with a bucket having primary and replicated OSDs.

Procedure

To perform a localized read, set rados_replica_read_policy to 'localize' in the OSD daemon configuration using the ceph config set command.
```
[ceph: root@host01 /]# ceph config set client.rgw.rgw.1 rados_replica_read_policy localize
```

Verification: Perform the below steps to verify the localized read from an OSD set.

Run the ceph osd tree command to view the OSDs and the data centers.

Example

[ceph: root@host01 /]# ceph osd tree

ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
-1         0.58557  root default
-3         0.29279      datacenter DC1
-2         0.09760          host ceph-ci-fbv67y-ammmck-node2
 2    hdd  0.02440              osd.2                             up   1.00000  1.00000
11    hdd  0.02440              osd.11                            up   1.00000  1.00000
17    hdd  0.02440              osd.17                            up   1.00000  1.00000
22    hdd  0.02440              osd.22                            up   1.00000  1.00000
-4         0.09760          host ceph-ci-fbv67y-ammmck-node3
 0    hdd  0.02440              osd.0                             up   1.00000  1.00000
 6    hdd  0.02440              osd.6                             up   1.00000  1.00000
12    hdd  0.02440              osd.12                            up   1.00000  1.00000
18    hdd  0.02440              osd.18                            up   1.00000  1.00000
-5         0.09760          host ceph-ci-fbv67y-ammmck-node4
 5    hdd  0.02440              osd.5                             up   1.00000  1.00000
10    hdd  0.02440              osd.10                            up   1.00000  1.00000
16    hdd  0.02440              osd.16                            up   1.00000  1.00000
23    hdd  0.02440              osd.23                            up   1.00000  1.00000
-7         0.29279      datacenter DC2
-6         0.09760          host ceph-ci-fbv67y-ammmck-node5
 3    hdd  0.02440              osd.3                             up   1.00000  1.00000
 8    hdd  0.02440              osd.8                             up   1.00000  1.00000
14    hdd  0.02440              osd.14                            up   1.00000  1.00000
20    hdd  0.02440              osd.20                            up   1.00000  1.00000
-8         0.09760          host ceph-ci-fbv67y-ammmck-node6
 4    hdd  0.02440              osd.4                             up   1.00000  1.00000
 9    hdd  0.02440              osd.9                             up   1.00000  1.00000
15    hdd  0.02440              osd.15                            up   1.00000  1.00000
21    hdd  0.02440              osd.21                            up   1.00000  1.00000
-9         0.09760          host ceph-ci-fbv67y-ammmck-node7
 1    hdd  0.02440              osd.1                             up   1.00000  1.00000
 7    hdd  0.02440              osd.7                             up   1.00000  1.00000
13    hdd  0.02440              osd.13                            up   1.00000  1.00000
19    hdd  0.02440              osd.19                            up   1.00000  1.00000

Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

Example

[ceph: root@host01 /]# ceph orch ps | grep rg

rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

Verify if a default read has happened by running the vim command on the Ceph Object Gateway logs.

Example

[ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log

2024-08-26T08:07:45.471+0000 7fc623e63640  1 ====== starting new request req=0x7fc5b93694a0 =====
2024-08-26T08:07:45.471+0000 7fc623e63640  1 -- 10.0.67.142:0/279982082 --> [v2:10.0.66.23:6816/73244434,v1:10.0.66.23:6817/73244434] -- osd_op(unknown.0.0:9081 11.55 11:ab26b168:::3acf4091-c54c-43b5-a495-c505fe545d25.27842.1_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+localize_reads+known_if_redirected+supports_pool_eio e3533) -- 0x55f781bd2000 con 0x55f77f0e8c00

You can see in the logs that a localized read has taken place.

Important

To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1

4.2.2. Performing balanced reads

You can perform a balanced read on a pool to retrieve evenly distributed OSDs across data centers. When a balanced READ is issued on a pool, the read operations are distributed evenly across all OSDs that are spread across the data centers.

Prerequisites

A stretch cluster with two data centers and Ceph Object Gateway configured on both.
A user created with a bucket and OSDs - primary and replicated OSDs.

Procedure

To perform a balanced read, set rados_replica_read_policy to 'balance' in the OSD daemon configuration using the ceph config set command.
```
[ceph: root@host01 /]# ceph config set client.rgw.rgw.1 rados_replica_read_policy balance
```

Verification: Perform the below steps to verify the balance read from an OSD set.

Run the ceph osd tree command to view the OSDs and the data centers.

Example

[ceph: root@host01 /]# ceph osd tree

ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
-1         0.58557  root default
-3         0.29279      datacenter DC1
-2         0.09760          host ceph-ci-fbv67y-ammmck-node2
 2    hdd  0.02440              osd.2                             up   1.00000  1.00000
11    hdd  0.02440              osd.11                            up   1.00000  1.00000
17    hdd  0.02440              osd.17                            up   1.00000  1.00000
22    hdd  0.02440              osd.22                            up   1.00000  1.00000
-4         0.09760          host ceph-ci-fbv67y-ammmck-node3
 0    hdd  0.02440              osd.0                             up   1.00000  1.00000
 6    hdd  0.02440              osd.6                             up   1.00000  1.00000
12    hdd  0.02440              osd.12                            up   1.00000  1.00000
18    hdd  0.02440              osd.18                            up   1.00000  1.00000
-5         0.09760          host ceph-ci-fbv67y-ammmck-node4
 5    hdd  0.02440              osd.5                             up   1.00000  1.00000
10    hdd  0.02440              osd.10                            up   1.00000  1.00000
16    hdd  0.02440              osd.16                            up   1.00000  1.00000
23    hdd  0.02440              osd.23                            up   1.00000  1.00000
-7         0.29279      datacenter DC2
-6         0.09760          host ceph-ci-fbv67y-ammmck-node5
 3    hdd  0.02440              osd.3                             up   1.00000  1.00000
 8    hdd  0.02440              osd.8                             up   1.00000  1.00000
14    hdd  0.02440              osd.14                            up   1.00000  1.00000
20    hdd  0.02440              osd.20                            up   1.00000  1.00000
-8         0.09760          host ceph-ci-fbv67y-ammmck-node6
 4    hdd  0.02440              osd.4                             up   1.00000  1.00000
 9    hdd  0.02440              osd.9                             up   1.00000  1.00000
15    hdd  0.02440              osd.15                            up   1.00000  1.00000
21    hdd  0.02440              osd.21                            up   1.00000  1.00000
-9         0.09760          host ceph-ci-fbv67y-ammmck-node7
 1    hdd  0.02440              osd.1                             up   1.00000  1.00000
 7    hdd  0.02440              osd.7                             up   1.00000  1.00000
13    hdd  0.02440              osd.13                            up   1.00000  1.00000
19    hdd  0.02440              osd.19                            up   1.00000  1.00000

Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

Example

[ceph: root@host01 /]# ceph orch ps | grep rg

rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

Verify if a balanced read has happened by running the vim command on the Ceph Object Gateway logs.

Example

[ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log

2024-08-27T09:32:25.510+0000 7f2a7a284640  1 ====== starting new request req=0x7f2a31fcf4a0 =====
2024-08-27T09:32:25.510+0000 7f2a7a284640  1 -- 10.0.67.142:0/3116867178 --> [v2:10.0.64.146:6816/2838383288,v1:10.0.64.146:6817/2838383288] -- osd_op(unknown.0.0:268731 11.55 11:ab26b168:::3acf4091-c54c-43b5-a495-c505fe545d25.27842.1_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+balance_reads+known_if_redirected+supports_pool_eio e3554) -- 0x55cd1b88dc00 con 0x55cd18dd6000

You can see in the logs that a balanced read has taken place.

Important

To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1

4.2.3. Performing default reads

You can perform a default read on a pool to retrieve data from primary data centers. When a default READ is issued on a pool, the IO operations are retrieved directly from each OSD in the data center.

Prerequisites

A stretch cluster with two data centers and Ceph Object Gateway configured on both.
A user created with a bucket and OSDs - primary and replicated OSDs.

Procedure

To perform a default read, set rados_replica_read_policy to 'default' in the OSD daemon configuration by using the ceph config set command.
Example
```
[ceph: root@host01 /]#ceph config set

client.rgw.rgw.1 advanced rados_replica_read_policy default
```
The IO operations from the closest OSD in a data center are retrieved when a GET operation is performed.

Verification: Perform the below steps to verify the localized read from an OSD set.

Run the ceph osd tree command to view the OSDs and the data centers.

Example

[ceph: root@host01 /]# ceph osd tree

ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
-1         0.58557  root default
-3         0.29279      datacenter DC1
-2         0.09760          host ceph-ci-fbv67y-ammmck-node2
 2    hdd  0.02440              osd.2                             up   1.00000  1.00000
11    hdd  0.02440              osd.11                            up   1.00000  1.00000
17    hdd  0.02440              osd.17                            up   1.00000  1.00000
22    hdd  0.02440              osd.22                            up   1.00000  1.00000
-4         0.09760          host ceph-ci-fbv67y-ammmck-node3
 0    hdd  0.02440              osd.0                             up   1.00000  1.00000
 6    hdd  0.02440              osd.6                             up   1.00000  1.00000
12    hdd  0.02440              osd.12                            up   1.00000  1.00000
18    hdd  0.02440              osd.18                            up   1.00000  1.00000
-5         0.09760          host ceph-ci-fbv67y-ammmck-node4
 5    hdd  0.02440              osd.5                             up   1.00000  1.00000
10    hdd  0.02440              osd.10                            up   1.00000  1.00000
16    hdd  0.02440              osd.16                            up   1.00000  1.00000
23    hdd  0.02440              osd.23                            up   1.00000  1.00000
-7         0.29279      datacenter DC2
-6         0.09760          host ceph-ci-fbv67y-ammmck-node5
 3    hdd  0.02440              osd.3                             up   1.00000  1.00000
 8    hdd  0.02440              osd.8                             up   1.00000  1.00000
14    hdd  0.02440              osd.14                            up   1.00000  1.00000
20    hdd  0.02440              osd.20                            up   1.00000  1.00000
-8         0.09760          host ceph-ci-fbv67y-ammmck-node6
 4    hdd  0.02440              osd.4                             up   1.00000  1.00000
 9    hdd  0.02440              osd.9                             up   1.00000  1.00000
15    hdd  0.02440              osd.15                            up   1.00000  1.00000
21    hdd  0.02440              osd.21                            up   1.00000  1.00000
-9         0.09760          host ceph-ci-fbv67y-ammmck-node7
 1    hdd  0.02440              osd.1                             up   1.00000  1.00000
 7    hdd  0.02440              osd.7                             up   1.00000  1.00000
13    hdd  0.02440              osd.13                            up   1.00000  1.00000
19    hdd  0.02440              osd.19                            up   1.00000  1.00000

Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

Example

ceph orch ps | grep rg

rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

Verify if a default read has happened by running the vim command on the Ceph Object Gateway logs.

Example

[ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log

2024-08-28T10:26:05.155+0000 7fe6b03dd640  1 ====== starting new request req=0x7fe6879674a0 =====
2024-08-28T10:26:05.156+0000 7fe6b03dd640  1 -- 10.0.64.251:0/2235882725 --> [v2:10.0.65.171:6800/4255735352,v1:10.0.65.171:6801/4255735352] -- osd_op(unknown.0.0:1123 11.6d 11:b69767fc:::699c2d80-5683-43c5-bdcd-e8912107c176.24827.3_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e4513) -- 0x5639da653800 con 0x5639d804d800

You can see in the logs that a default read has taken place.

Important

To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1

[ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1

Chapter 4. Stretch clusters for Ceph storage

4.1. Stretch mode for a storage cluster

4.1.1. Setting the CRUSH location for the daemons

4.1.2. Entering the stretch mode

4.1.3. Adding OSD hosts in stretch mode

4.2. Read affinity in stretch clusters

4.2.1. Performing localized reads

4.2.2. Performing balanced reads

4.2.3. Performing default reads

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links