Questo contenuto non è disponibile nella lingua selezionata.

Chapter 4. Stretch clusters for Ceph storage


As a storage administrator, you can configure stretch clusters by entering stretch mode with 2-site clusters.

Red Hat Ceph Storage is capable of withstanding the loss of Ceph OSDs because of its network and cluster, which are equally reliable with failures randomly distributed across the CRUSH map. If a number of OSDs is shut down, the remaining OSDs and monitors still manage to operate.

However, this might not be the best solution for some stretched cluster configurations where a significant part of the Ceph cluster can use only a single network component. The example is a single cluster located in multiple data centers, for which the user wants to sustain a loss of a full data center.

The standard configuration is with two data centers. Other configurations are in clouds or availability zones. Each site holds two copies of the data, therefore, the replication size is four. The third site should have a tiebreaker monitor, this can be a virtual machine or high-latency compared to the main sites. This monitor chooses one of the sites to restore data if the network connection fails and both data centers remain active.

Note

The standard Ceph configuration survives many failures of the network or data centers and it never compromises data consistency. If you restore enough Ceph servers following a failure, it recovers. Ceph maintains availability if you lose a data center, but can still form a quorum of monitors and have all the data available with enough copies to satisfy pools’ min_size, or CRUSH rules that replicate again to meet the size.

Note

There are no additional steps to power down a stretch cluster. You can see the Powering down and rebooting Red Hat Ceph Storage cluster for more information.

Stretch cluster failures

Red Hat Ceph Storage never compromises on data integrity and consistency. If there is a network failure or a loss of nodes and the services can still be restored, Ceph returns to normal functionality on its own.

However, there are situations where you lose data availability even if you have enough servers available to meet Ceph’s consistency and sizing constraints, or where you unexpectedly do not meet the constraints.

First important type of failure is caused by inconsistent networks. If there is a network split, Ceph might be unable to mark OSD as down to remove it from the acting placement group (PG) sets despite the primary OSD being unable to replicate data. When this happens, the I/O is not permitted because Ceph cannot meet its durability guarantees.

The second important category of failures is when it appears that you have data replicated across data enters, but the constraints are not sufficient to guarantee this. For example, you might have data centers A and B, and the CRUSH rule targets three copies and places a copy in each data center with a min_size of 2. The PG might go active with two copies in site A and no copies in site B, which means that if you lose site A, you lose the data and Ceph cannot operate on it. This situation is difficult to avoid with standard CRUSH rules.

4.1. Stretch mode for a storage cluster

To configure stretch clusters, you must enter the stretch mode. When stretch mode is enabled, the Ceph OSDs only take PGs as active when they peer across data centers, or whichever other CRUSH bucket type you specified, assuming both are active. Pools increase in size from the default three to four, with two copies on each site.

In stretch mode, Ceph OSDs are only allowed to connect to monitors within the same data center. New monitors are not allowed to join the cluster without specified location.

If all the OSDs and monitors from a data center become inaccessible at once, the surviving data center will enter a degraded stretch mode. This issues a warning, reduces the min_size to 1, and allows the cluster to reach an active state with the data from the remaining site.

Note

The degraded state also triggers warnings that the pools are too small, because the pool size does not get changed. However, a special stretch mode flag prevents the OSDs from creating extra copies in the remaining data center, therefore it still keeps 2 copies.

When the missing data center becomes accesible again, the cluster enters recovery stretch mode. This changes the warning and allows peering, but still requires only the OSDs from the data center, which was up the whole time.

When all PGs are in a known state and are not degraded or incomplete, the cluster goes back to the regular stretch mode, ends the warning, and restores min_size to its starting value 2. The cluster again requires both sites to peer, not only the site that stayed up the whole time, therefore you can fail over to the other site, if necessary.

Stretch mode limitations

  • It is not possible to exit from stretch mode once it is entered.
  • You cannot use erasure-coded pools with clusters in stretch mode. You can neither enter the stretch mode with erasure-coded pools, nor create an erasure-coded pool when the stretch mode is active.
  • Stretch mode with no more than two sites is supported.
  • The weights of the two sites should be the same. If they are not, you receive the following error:

    Example

    [ceph: root@host01 /]# ceph mon enable_stretch_mode host05 stretch_rule datacenter
    
    Error EINVAL: the 2 datacenter instances in the cluster have differing weights 25947 and 15728 but stretch mode currently requires they be the same!

    To achieve same weights on both sites, the Ceph OSDs deployed in the two sites should be of equal size, that is, storage capacity in the first site is equivalent to storage capacity in the second site.

  • While it is not enforced, you should run two Ceph monitors on each site and a tiebreaker, for a total of five. This is because OSDs can only connect to monitors in their own site when in stretch mode.
  • You have to create your own CRUSH rule, which provides two copies on each site, which totals to four on both sites.
  • You cannot enable stretch mode if you have existing pools with non-default size or min_size.
  • Because the cluster runs with min_size 1 when degraded, you should only use stretch mode with all-flash OSDs. This minimizes the time needed to recover once connectivity is restored, and minimizes the potential for data loss.

Additional Resources

4.1.1. Setting the CRUSH location for the daemons

Before you enter the stretch mode, you need to prepare the cluster by setting the CRUSH location to the daemons in the Red Hat Ceph Storage cluster. There are two ways to do this:

  • Bootstrap the cluster through a service configuration file, where the locations are added to the hosts as part of deployment.
  • Set the locations manually through ceph osd crush add-bucket and ceph osd crush move commands after the cluster is deployed.

Method 1: Bootstrapping the cluster

Prerequisites

  • Root-level access to the nodes.

Procedure

  1. If you are bootstrapping your new storage cluster, you can create the service configuration .yaml file that adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services should run:

    Example

    service_type: host
    addr: host01
    hostname: host01
    location:
      root: default
      datacenter: DC1
    labels:
      - osd
      - mon
      - mgr
    ---
    service_type: host
    addr: host02
    hostname: host02
    location:
      datacenter: DC1
    labels:
      - osd
      - mon
    ---
    service_type: host
    addr: host03
    hostname: host03
    location:
      datacenter: DC1
    labels:
      - osd
      - mds
      - rgw
    ---
    service_type: host
    addr: host04
    hostname: host04
    location:
      root: default
      datacenter: DC2
    labels:
      - osd
      - mon
      - mgr
    ---
    service_type: host
    addr: host05
    hostname: host05
    location:
      datacenter: DC2
    labels:
      - osd
      - mon
    ---
    service_type: host
    addr: host06
    hostname: host06
    location:
      datacenter: DC2
    labels:
      - osd
      - mds
      - rgw
    ---
    service_type: host
    addr: host07
    hostname: host07
    labels:
      - mon
    ---
    service_type: mon
    placement:
      label: "mon"
    ---
    service_id: cephfs
    placement:
      label: "mds"
    ---
    service_type: mgr
    service_name: mgr
    placement:
      label: "mgr"
    ---
    service_type: osd
    service_id: all-available-devices
    service_name: osd.all-available-devices
    placement:
      label: "osd"
    spec:
      data_devices:
        all: true
    ---
    service_type: rgw
    service_id: objectgw
    service_name: rgw.objectgw
    placement:
      count: 2
      label: "rgw"
    spec:
      rgw_frontend_port: 8080

  2. Bootstrap the storage cluster with the --apply-spec option:

    Syntax

    cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

    Example

    [root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

    Important

    You can use different command options with the cephadm bootstrap command. However, always include the --apply-spec option to use the service configuration file and configure the host locations.

Additional Resources

Method 2: Setting the locations after the deployment

Prerequisites

  • Root-level access to the nodes.

Procedure

  1. Add two buckets to which you plan to set the location of your non-tiebreaker monitors to the CRUSH map, specifying the bucket type as as datacenter:

    Syntax

    ceph osd crush add-bucket BUCKET_NAME BUCKET_TYPE

    Example

    [ceph: root@host01 /]# ceph osd crush add-bucket DC1 datacenter
    [ceph: root@host01 /]# ceph osd crush add-bucket DC2 datacenter

  2. Move the buckets under root=default:

    Syntax

    ceph osd crush move BUCKET_NAME root=default

    Example

    [ceph: root@host01 /]# ceph osd crush move DC1 root=default
    [ceph: root@host01 /]# ceph osd crush move DC2 root=default

  3. Move the OSD hosts according to the required CRUSH placement:

    Syntax

    ceph osd crush move HOST datacenter=DATACENTER

    Example

    [ceph: root@host01 /]# ceph osd crush move host01 datacenter=DC1

4.1.2. Entering the stretch mode

The new stretch mode is designed to handle two sites. There is a lower risk of component availability outages with 2-site clusters.

Prerequisites

  • Root-level access to the nodes.
  • The CRUSH location is set to the hosts.

Procedure

  1. Set the location of each monitor, matching your CRUSH map:

    Syntax

    ceph mon set_location HOST datacenter=DATACENTER

    Example

    [ceph: root@host01 /]# ceph mon set_location host01 datacenter=DC1
    [ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC1
    [ceph: root@host01 /]# ceph mon set_location host04 datacenter=DC2
    [ceph: root@host01 /]# ceph mon set_location host05 datacenter=DC2
    [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3

  2. Generate a CRUSH rule which places two copies on each data center:

    Syntax

    ceph osd getcrushmap > COMPILED_CRUSHMAP_FILENAME
    crushtool -d COMPILED_CRUSHMAP_FILENAME -o DECOMPILED_CRUSHMAP_FILENAME

    Example

    [ceph: root@host01 /]# ceph osd getcrushmap > crush.map.bin
    [ceph: root@host01 /]# crushtool -d crush.map.bin -o crush.map.txt

    1. Edit the decompiled CRUSH map file to add a new rule:

      Example

      rule stretch_rule {
              id 1 1
              type replicated
              min_size 1
              max_size 10
              step take DC1 2
              step chooseleaf firstn 2 type host
              step emit
              step take DC2 3
              step chooseleaf firstn 2 type host
              step emit
      }

      1
      The rule id has to be unique. In this example, there is only one more rule with id 0, thereby the id 1 is used, however you might need to use a different rule ID depending on the number of existing rules.
      2 3
      In this example, there are two data center buckets named DC1 and DC2.
      Note

      This rule makes the cluster have read-affinity towards data center DC1. Therefore, all the reads or writes happen through Ceph OSDs placed in DC1.

      If this is not desirable, and reads or writes are to be distributed evenly across the zones, the CRUSH rule is the following:

      Example

      rule stretch_rule {
      id 1
      type replicated
      min_size 1
      max_size 10
      step take default
      step choose firstn 0 type datacenter
      step chooseleaf firstn 2 type host
      step emit
      }

      In this rule, the data center is selected randomly and automatically.

      See CRUSH rules for more information on firstn and indep options.

  3. Inject the CRUSH map to make the rule available to the cluster:

    Syntax

    crushtool -c DECOMPILED_CRUSHMAP_FILENAME -o COMPILED_CRUSHMAP_FILENAME
    ceph osd setcrushmap -i COMPILED_CRUSHMAP_FILENAME

    Example

    [ceph: root@host01 /]# crushtool -c crush.map.txt -o crush2.map.bin
    [ceph: root@host01 /]# ceph osd setcrushmap -i crush2.map.bin

  4. If you do not run the monitors in connectivity mode, set the election strategy to connectivity:

    Example

    [ceph: root@host01 /]# ceph mon set election_strategy connectivity

  5. Enter stretch mode by setting the location of the tiebreaker monitor to split across the data centers:

    Syntax

    ceph mon set_location HOST datacenter=DATACENTER
    ceph mon enable_stretch_mode HOST stretch_rule datacenter

    Example

    [ceph: root@host01 /]# ceph mon set_location host07 datacenter=DC3
    [ceph: root@host01 /]# ceph mon enable_stretch_mode host07 stretch_rule datacenter

    In this example the monitor mon.host07 is the tiebreaker.

    Important

    The location of the tiebreaker monitor should differ from the data centers to which you previously set the non-tiebreaker monitors. In the example above, it is data center DC3.

    Important

    Do not add this data center to the CRUSH map as it results in the following error when you try to enter stretch mode:

    Error EINVAL: there are 3 datacenters in the cluster but stretch mode currently only works with 2!
    Note

    If you are writing your own tooling for deploying Ceph, you can use a new --set-crush-location option when booting monitors, instead of running the ceph mon set_location command. This option accepts only a single bucket=location pair, for example ceph-mon --set-crush-location 'datacenter=DC1', which must match the bucket type you specified when running the enable_stretch_mode command.

  6. Verify that the stretch mode is enabled successfully:

    Example

    [ceph: root@host01 /]# ceph osd dump
    
    epoch 361
    fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
    created 2023-01-16T05:47:28.482717+0000
    modified 2023-01-17T17:36:50.066183+0000
    flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
    crush_version 31
    full_ratio 0.95
    backfillfull_ratio 0.92
    nearfull_ratio 0.85
    require_min_compat_client luminous
    min_compat_client luminous
    require_osd_release quincy
    stretch_mode_enabled true
    stretch_bucket_count 2
    degraded_stretch_mode 0
    recovering_stretch_mode 0
    stretch_mode_bucket 8

    The stretch_mode_enabled should be set to true. You can also see the number of stretch buckets, stretch mode buckets, and if the stretch mode is degraded or recovering.

  7. Verify that the monitors are in an appropriate locations:

    Example

    [ceph: root@host01 /]# ceph mon dump
    
    epoch 19
    fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
    last_changed 2023-01-17T04:12:05.709475+0000
    created 2023-01-16T05:47:25.631684+0000
    min_mon_release 16 (pacific)
    election_strategy: 3
    stretch_mode_enabled 1
    tiebreaker_mon host07
    disallowed_leaders host07
    0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host07; crush_location {datacenter=DC3}
    1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
    2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
    3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
    4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
    dumped monmap epoch 19

    You can also see which monitor is the tiebreaker, and the monitor election strategy.

Additional Resources

4.1.3. Adding OSD hosts in stretch mode

You can add Ceph OSDs in the stretch mode. The procedure is similar to the addition of the OSD hosts on a cluster where stretch mode is not enabled.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Stretch mode in enabled on a cluster.
  • Root-level access to the nodes.

Procedure

  1. List the available devices to deploy OSDs:

    Syntax

    ceph orch device ls [--hostname=HOST_1 HOST_2] [--wide] [--refresh]

    Example

    [ceph: root@host01 /]# ceph orch device ls

  2. Deploy the OSDs on specific hosts or on all the available devices:

    • Create an OSD from a specific device on a specific host:

      Syntax

      ceph orch daemon add osd HOST:DEVICE_PATH

      Example

      [ceph: root@host01 /]# ceph orch daemon add osd host03:/dev/sdb

    • Deploy OSDs on any available and unused devices:

      Important

      This command creates collocated WAL and DB devices. If you want to create non-collocated devices, do not use this command.

      Example

      [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

  3. Move the OSD hosts under the CRUSH bucket:

    Syntax

    ceph osd crush move HOST datacenter=DATACENTER

    Example

    [ceph: root@host01 /]# ceph osd crush move host03 datacenter=DC1
    [ceph: root@host01 /]# ceph osd crush move host06 datacenter=DC2

    Note

    Ensure you add the same topology nodes on both sites. Issues might arise if hosts are added only on one site.

Additional Resources

  • See Adding OSDs for more information about the addition of Ceph OSDs.

4.2. Read affinity in stretch clusters

Read Affinity reduces cross-zone traffic by keeping the data access within the respective data centers.

For stretched clusters deployed in multi-zone environments, the read affinity topology implementation provides a mechanism to help keep traffic within the data center it originated from. Ceph Object Gateway volumes have the ability to read data from an OSD in proximity to the client, according to OSD locations defined in the CRUSH map and topology labels on nodes.

For example, a stretch cluster contains a Ceph Object Gateway primary OSD and replicated OSDs spread across two data centers A and B. If a GET action is performed on an Object in data center A, the READ operation is performed on the data of the OSDs closest to the client in data center A.

4.2.1. Performing localized reads

You can perform a localized read on a replicated pool in a stretch cluster. When a localized read request is made on a replicated pool, Ceph selects the local OSDs closest to the client based on the client location specified in crush_location.

Prerequisites

  • A stretch cluster with two data centers and Ceph Object Gateway configured on both.
  • A user created with a bucket having primary and replicated OSDs.

Procedure

  • To perform a localized read, set rados_replica_read_policy to 'localize' in the OSD daemon configuration using the ceph config set command.

    [ceph: root@host01 /]# ceph config set client.rgw.rgw.1 rados_replica_read_policy localize
  • Verification: Perform the below steps to verify the localized read from an OSD set.

    1. Run the ceph osd tree command to view the OSDs and the data centers.

      Example

      [ceph: root@host01 /]# ceph osd tree
      
      ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
      -1         0.58557  root default
      -3         0.29279      datacenter DC1
      -2         0.09760          host ceph-ci-fbv67y-ammmck-node2
       2    hdd  0.02440              osd.2                             up   1.00000  1.00000
      11    hdd  0.02440              osd.11                            up   1.00000  1.00000
      17    hdd  0.02440              osd.17                            up   1.00000  1.00000
      22    hdd  0.02440              osd.22                            up   1.00000  1.00000
      -4         0.09760          host ceph-ci-fbv67y-ammmck-node3
       0    hdd  0.02440              osd.0                             up   1.00000  1.00000
       6    hdd  0.02440              osd.6                             up   1.00000  1.00000
      12    hdd  0.02440              osd.12                            up   1.00000  1.00000
      18    hdd  0.02440              osd.18                            up   1.00000  1.00000
      -5         0.09760          host ceph-ci-fbv67y-ammmck-node4
       5    hdd  0.02440              osd.5                             up   1.00000  1.00000
      10    hdd  0.02440              osd.10                            up   1.00000  1.00000
      16    hdd  0.02440              osd.16                            up   1.00000  1.00000
      23    hdd  0.02440              osd.23                            up   1.00000  1.00000
      -7         0.29279      datacenter DC2
      -6         0.09760          host ceph-ci-fbv67y-ammmck-node5
       3    hdd  0.02440              osd.3                             up   1.00000  1.00000
       8    hdd  0.02440              osd.8                             up   1.00000  1.00000
      14    hdd  0.02440              osd.14                            up   1.00000  1.00000
      20    hdd  0.02440              osd.20                            up   1.00000  1.00000
      -8         0.09760          host ceph-ci-fbv67y-ammmck-node6
       4    hdd  0.02440              osd.4                             up   1.00000  1.00000
       9    hdd  0.02440              osd.9                             up   1.00000  1.00000
      15    hdd  0.02440              osd.15                            up   1.00000  1.00000
      21    hdd  0.02440              osd.21                            up   1.00000  1.00000
      -9         0.09760          host ceph-ci-fbv67y-ammmck-node7
       1    hdd  0.02440              osd.1                             up   1.00000  1.00000
       7    hdd  0.02440              osd.7                             up   1.00000  1.00000
      13    hdd  0.02440              osd.13                            up   1.00000  1.00000
      19    hdd  0.02440              osd.19                            up   1.00000  1.00000

    2. Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

      Example

      [ceph: root@host01 /]# ceph orch ps | grep rg
      
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

    3. Verify if a default read has happened by running the vim command on the Ceph Object Gateway logs.

      Example

      [ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log
      
      2024-08-26T08:07:45.471+0000 7fc623e63640  1 ====== starting new request req=0x7fc5b93694a0 =====
      2024-08-26T08:07:45.471+0000 7fc623e63640  1 -- 10.0.67.142:0/279982082 --> [v2:10.0.66.23:6816/73244434,v1:10.0.66.23:6817/73244434] -- osd_op(unknown.0.0:9081 11.55 11:ab26b168:::3acf4091-c54c-43b5-a495-c505fe545d25.27842.1_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+localize_reads+known_if_redirected+supports_pool_eio e3533) -- 0x55f781bd2000 con 0x55f77f0e8c00

      You can see in the logs that a localized read has taken place.

      Important

      To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1
      
      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1

4.2.2. Performing balanced reads

You can perform a balanced read on a pool to retrieve evenly distributed OSDs across data centers. When a balanced READ is issued on a pool, the read operations are distributed evenly across all OSDs that are spread across the data centers.

Prerequisites

  • A stretch cluster with two data centers and Ceph Object Gateway configured on both.
  • A user created with a bucket and OSDs - primary and replicated OSDs.

Procedure

  • To perform a balanced read, set rados_replica_read_policy to 'balance' in the OSD daemon configuration using the ceph config set command.

    [ceph: root@host01 /]# ceph config set client.rgw.rgw.1 rados_replica_read_policy balance
  • Verification: Perform the below steps to verify the balance read from an OSD set.

    1. Run the ceph osd tree command to view the OSDs and the data centers.

      Example

      [ceph: root@host01 /]# ceph osd tree
      
      ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
      -1         0.58557  root default
      -3         0.29279      datacenter DC1
      -2         0.09760          host ceph-ci-fbv67y-ammmck-node2
       2    hdd  0.02440              osd.2                             up   1.00000  1.00000
      11    hdd  0.02440              osd.11                            up   1.00000  1.00000
      17    hdd  0.02440              osd.17                            up   1.00000  1.00000
      22    hdd  0.02440              osd.22                            up   1.00000  1.00000
      -4         0.09760          host ceph-ci-fbv67y-ammmck-node3
       0    hdd  0.02440              osd.0                             up   1.00000  1.00000
       6    hdd  0.02440              osd.6                             up   1.00000  1.00000
      12    hdd  0.02440              osd.12                            up   1.00000  1.00000
      18    hdd  0.02440              osd.18                            up   1.00000  1.00000
      -5         0.09760          host ceph-ci-fbv67y-ammmck-node4
       5    hdd  0.02440              osd.5                             up   1.00000  1.00000
      10    hdd  0.02440              osd.10                            up   1.00000  1.00000
      16    hdd  0.02440              osd.16                            up   1.00000  1.00000
      23    hdd  0.02440              osd.23                            up   1.00000  1.00000
      -7         0.29279      datacenter DC2
      -6         0.09760          host ceph-ci-fbv67y-ammmck-node5
       3    hdd  0.02440              osd.3                             up   1.00000  1.00000
       8    hdd  0.02440              osd.8                             up   1.00000  1.00000
      14    hdd  0.02440              osd.14                            up   1.00000  1.00000
      20    hdd  0.02440              osd.20                            up   1.00000  1.00000
      -8         0.09760          host ceph-ci-fbv67y-ammmck-node6
       4    hdd  0.02440              osd.4                             up   1.00000  1.00000
       9    hdd  0.02440              osd.9                             up   1.00000  1.00000
      15    hdd  0.02440              osd.15                            up   1.00000  1.00000
      21    hdd  0.02440              osd.21                            up   1.00000  1.00000
      -9         0.09760          host ceph-ci-fbv67y-ammmck-node7
       1    hdd  0.02440              osd.1                             up   1.00000  1.00000
       7    hdd  0.02440              osd.7                             up   1.00000  1.00000
      13    hdd  0.02440              osd.13                            up   1.00000  1.00000
      19    hdd  0.02440              osd.19                            up   1.00000  1.00000

    2. Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

      Example

      [ceph: root@host01 /]# ceph orch ps | grep rg
      
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

    3. Verify if a balanced read has happened by running the vim command on the Ceph Object Gateway logs.

      Example

      [ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log
      
      2024-08-27T09:32:25.510+0000 7f2a7a284640  1 ====== starting new request req=0x7f2a31fcf4a0 =====
      2024-08-27T09:32:25.510+0000 7f2a7a284640  1 -- 10.0.67.142:0/3116867178 --> [v2:10.0.64.146:6816/2838383288,v1:10.0.64.146:6817/2838383288] -- osd_op(unknown.0.0:268731 11.55 11:ab26b168:::3acf4091-c54c-43b5-a495-c505fe545d25.27842.1_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+balance_reads+known_if_redirected+supports_pool_eio e3554) -- 0x55cd1b88dc00 con 0x55cd18dd6000

      You can see in the logs that a balanced read has taken place.

      Important

      To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1
      
      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1

4.2.3. Performing default reads

You can perform a default read on a pool to retrieve data from primary data centers. When a default READ is issued on a pool, the IO operations are retrieved directly from each OSD in the data center.

Prerequisites

  • A stretch cluster with two data centers and Ceph Object Gateway configured on both.
  • A user created with a bucket and OSDs - primary and replicated OSDs.

Procedure

  • To perform a default read, set rados_replica_read_policy to 'default' in the OSD daemon configuration by using the ceph config set command.

    Example

    [ceph: root@host01 /]#ceph config set
    
    client.rgw.rgw.1 advanced rados_replica_read_policy default

    The IO operations from the closest OSD in a data center are retrieved when a GET operation is performed.

  • Verification: Perform the below steps to verify the localized read from an OSD set.

    1. Run the ceph osd tree command to view the OSDs and the data centers.

      Example

      [ceph: root@host01 /]# ceph osd tree
      
      ID  CLASS  WEIGHT   TYPE NAME                                 STATUS  REWEIGHT  PRI-AFF
      -1         0.58557  root default
      -3         0.29279      datacenter DC1
      -2         0.09760          host ceph-ci-fbv67y-ammmck-node2
       2    hdd  0.02440              osd.2                             up   1.00000  1.00000
      11    hdd  0.02440              osd.11                            up   1.00000  1.00000
      17    hdd  0.02440              osd.17                            up   1.00000  1.00000
      22    hdd  0.02440              osd.22                            up   1.00000  1.00000
      -4         0.09760          host ceph-ci-fbv67y-ammmck-node3
       0    hdd  0.02440              osd.0                             up   1.00000  1.00000
       6    hdd  0.02440              osd.6                             up   1.00000  1.00000
      12    hdd  0.02440              osd.12                            up   1.00000  1.00000
      18    hdd  0.02440              osd.18                            up   1.00000  1.00000
      -5         0.09760          host ceph-ci-fbv67y-ammmck-node4
       5    hdd  0.02440              osd.5                             up   1.00000  1.00000
      10    hdd  0.02440              osd.10                            up   1.00000  1.00000
      16    hdd  0.02440              osd.16                            up   1.00000  1.00000
      23    hdd  0.02440              osd.23                            up   1.00000  1.00000
      -7         0.29279      datacenter DC2
      -6         0.09760          host ceph-ci-fbv67y-ammmck-node5
       3    hdd  0.02440              osd.3                             up   1.00000  1.00000
       8    hdd  0.02440              osd.8                             up   1.00000  1.00000
      14    hdd  0.02440              osd.14                            up   1.00000  1.00000
      20    hdd  0.02440              osd.20                            up   1.00000  1.00000
      -8         0.09760          host ceph-ci-fbv67y-ammmck-node6
       4    hdd  0.02440              osd.4                             up   1.00000  1.00000
       9    hdd  0.02440              osd.9                             up   1.00000  1.00000
      15    hdd  0.02440              osd.15                            up   1.00000  1.00000
      21    hdd  0.02440              osd.21                            up   1.00000  1.00000
      -9         0.09760          host ceph-ci-fbv67y-ammmck-node7
       1    hdd  0.02440              osd.1                             up   1.00000  1.00000
       7    hdd  0.02440              osd.7                             up   1.00000  1.00000
      13    hdd  0.02440              osd.13                            up   1.00000  1.00000
      19    hdd  0.02440              osd.19                            up   1.00000  1.00000

    2. Run the ceph orch command to identify the Ceph Object Gateway daemons in the data centers.

      Example

      ceph orch ps | grep rg
      
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node4.dmsmex         ceph-ci-fbv67y-ammmck-node4            *:80              running (4h)     10m ago  22h    93.3M        -  19.1.0-55.el9cp  0ee0a0ad94c7  34f27723ccd2
      rgw.rgw.1.ceph-ci-fbv67y-ammmck-node7.pocecp         ceph-ci-fbv67y-ammmck-node7            *:80              running (4h)     10m ago  22h    96.4M        -  19.1.0-55.el9cp  0ee0a0ad94c7  40e4f2a6d4c4

    3. Verify if a default read has happened by running the vim command on the Ceph Object Gateway logs.

      Example

      [ceph: root@host01 /]# vim /var/log/ceph/<fsid>/<ceph-client-rgw>.log
      
      2024-08-28T10:26:05.155+0000 7fe6b03dd640  1 ====== starting new request req=0x7fe6879674a0 =====
      2024-08-28T10:26:05.156+0000 7fe6b03dd640  1 -- 10.0.64.251:0/2235882725 --> [v2:10.0.65.171:6800/4255735352,v1:10.0.65.171:6801/4255735352] -- osd_op(unknown.0.0:1123 11.6d 11:b69767fc:::699c2d80-5683-43c5-bdcd-e8912107c176.24827.3_f1:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e4513) -- 0x5639da653800 con 0x5639d804d800

      You can see in the logs that a default read has taken place.

      Important

      To be able to view the debug logs, you must first enable debug_ms 1 in the configuration by running the ceph config set command.

      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node4.dgvrmx    advanced  debug_ms    1/1
      
      [ceph: root@host01 /]#ceph config set client.rgw.rgw.1.ceph-ci-gune2w-mysx73-node7.rfkqqq    advanced  debug_ms    1/1
Red Hat logoGithubRedditYoutubeTwitter

Formazione

Prova, acquista e vendi

Community

Informazioni sulla documentazione di Red Hat

Aiutiamo gli utenti Red Hat a innovarsi e raggiungere i propri obiettivi con i nostri prodotti e servizi grazie a contenuti di cui possono fidarsi.

Rendiamo l’open source più inclusivo

Red Hat si impegna a sostituire il linguaggio problematico nel codice, nella documentazione e nelle proprietà web. Per maggiori dettagli, visita ilBlog di Red Hat.

Informazioni su Red Hat

Forniamo soluzioni consolidate che rendono più semplice per le aziende lavorare su piattaforme e ambienti diversi, dal datacenter centrale all'edge della rete.

© 2024 Red Hat, Inc.