Chapter 5. Multi-site configuration and administration

Procedure

1. Set the <ArchiveZone> lifecycle policy rule. For more information about creating a lifecycle policy, see See the Creating a lifecycle management policy section in the Red Hat Ceph Storage Object Gateway Guide for more details.

   Example

   <?xml version="1.0" ?>
   <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
           <Rule>
                   <ID>delete-1-days-az</ID>
                   <Filter>
   		  <Prefix></Prefix>
   		  <ArchiveZone />   1
                   </Filter>
                   <Status>Enabled</Status>
                   <Expiration>
                           <Days>1</Days>
                   </Expiration>
           </Rule>
   </LifecycleConfiguration>

2. Optional: See if a specific lifecycle policy contains an archive zone rule.

   Syntax

   radosgw-admin lc get --bucket BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin lc get --bucket test-bkt
   
   {
       "prefix_map": {
           "": {
               "status": true,
               "dm_expiration": true,
               "expiration": 0,
               "noncur_expiration": 2,
               "mp_expiration": 0,
               "transitions": {},
               "noncur_transitions": {}
           }
       },
       "rule_map": [
           {
               "id": "Rule 1",
               "rule": {
                   "id": "Rule 1",
                   "prefix": "",
                   "status": "Enabled",
                   "expiration": {
                       "days": "",
                       "date": ""
                   },
                   "noncur_expiration": {
                       "days": "2",
                       "date": ""
                   },
                   "mp_expiration": {
                       "days": "",
                       "date": ""
                   },
                   "filter": {
                       "prefix": "",
                       "obj_tags": {
                           "tagset": {}
                       },
                       "archivezone": ""   1
                   },
                   "transitions": {},
                   "noncur_transitions": {},
                   "dm_expiration": true
               }
           }
       ]
   }

   1 1

   The archive zone rule. This is an example of a lifecycle policy with an archive zone rule.

3. If the Ceph Object Gateway user is deleted, the buckets at the archive site owned by that user is inaccessible. Link those buckets to another Ceph Object Gateway user to access the data.

   Syntax

    radosgw-admin bucket link --uid NEW_USER_ID --bucket BUCKET_NAME --yes-i-really-mean-it

   Example

   [ceph: root@host01 /]# radosgw-admin bucket link --uid arcuser1 --bucket arc1-deleted-da473fbbaded232dc5d1e434675c1068 --yes-i-really-mean-it

Procedure

1. Create a sync policy group or a bucket policy. To create a bucket policy, use the --bucket option:

   Syntax

   radosgw-admin sync group create --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled | allowed | forbidden

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --group-id=mygroup1 --status=enabled

2. Optional: Manually complete the sync process for bucket granular replication.

   Note

   This step is mandatory when using as part of an archive zone with bucket granular replication if the policy has data written or the bucket was resharded when the policy was forbidden.

   Syntax

   radosgw-admin bucket sync run

   Example

   [ceph: root@host01 /]# radosgw-admin bucket sync run

Procedure

1. Modify the sync policy group or a bucket policy. To modify a bucket policy, use the --bucket option.

   Syntax

   radosgw-admin sync group modify --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled | allowed | forbidden

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id=mygroup1 --status=forbidden

2. Optional: Manually complete the sync process for bucket granular replication.

   Note

   This step is mandatory when using as part of an archive zone with bucket granular replication if the policy has data written or the bucket was resharded when the policy was forbidden.

   Syntax

   radosgw-admin bucket sync run

   Example

   [ceph: root@host01 /]# radosgw-admin bucket sync run

Procedure

1. Create or update a directional sync flow. To create or update directional sync flow for a specific bucket, use the --bucket option.

   Syntax

   radosgw-admin sync group flow create --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE --dest-zone=DESTINATION_ZONE

2. Create or update a symmetrical sync flow. To specify multiple zones for a symmetrical flow type, use a comma-separated list for the --zones option.

   Syntax

   radosgw-admin sync group flow create --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME1,ZONE_NAME2

   zones are comma-separated lists of all zones that need to be added to the flow.

Procedure

1. Remove a directional sync flow. To remove the directional sync flow for a specific bucket, use the --bucket option.

   Syntax

   radosgw-admin sync group flow remove --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE --dest-zone=DESTINATION_ZONE

2. Remove specific zones from a symmetrical sync flow. To remove multiple zones from a symmetrical flow, use a comma-separated list for the --zones option.

   Syntax

   radosgw-admin sync group flow remove --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME1,ZONE_NAME2

3. Remove a symmetrical sync flow. To remove the sync flow at the zonegroup level, remove the --bucket option.

   Syntax

   radosgw-admin sync group flow remove --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME1,ZONE_NAME2

Procedure

1. Create a new sync group with the status set to allowed.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --group-id=group1 --status=allowed

   Note

   Until a fully configured zonegroup replication policy is created, it is recommended to set the --status to allowed, to prevent the replication from starting.

2. Create a flow policy for the newly created group with the --flow-type set as symmetrical to enable bi-directional replication.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group flow create --group-id=group1 \
                           --flow-id=flow-mirror --flow-type=symmetrical \
                           --zones=us-east,us-west

3. Create a new pipe called pipe.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --group-id=group1 \
                             --pipe-id=pipe1 --source-zones='*' \
                             --source-bucket='*' --dest-zones='*' \
                             --dest-bucket='*'

   Note

   Use the * wildcard for zones to include all zones set in the previous flow policy, and * for buckets to replicate all existing buckets in the zones.

4. After configuring the bucket sync policy, set the --status to enabled.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id=group1 --status=enabled

5. Update and commit the new period.

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

   Note

   Updating and committing the period is mandatory for a zonegroup policy.

6. Optional: Check the sync source, and destination for a specific bucket. All buckets in zones us-east and us-west replicates bi-directionally.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info -bucket buck
   {
       "sources": [
           {
               "id": "pipe1",
               "source": {
                   "zone": "us-east",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               "dest": {
                   "zone": "us-west",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               ...
           }
       ],
       "dests": [
           {
               "id": "pipe1",
               "source": {
                   "zone": "us-west",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               "dest": {
                   "zone": "us-east",
                   "bucket": "buck:115b12b3-....4409.1"
               },
              ...
           }
       ],
       ...
   }

   The id field in the above output reflects the pipe rule that generated that entry. A single rule can generate multiple sync entries as seen in the below example.

Procedure

1. On the primary zone, create a new sync group with the status set to allowed.

   Syntax

   radosgw-admin sync group create --group-id=GROUP_ID --status=allowed

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --group-id=group1 --status=allowed

   Note

   Until a fully configured zonegroup replication policy is created, it is recommended to set the --status to allowed, to prevent the replication from starting.

2. Create the flow.

   Syntax

   radosgw-admin sync group flow create --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE_NAME --dest-zone=DESTINATION_ZONE_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync group flow create --group-id=group1 --flow-id=us-west-backup --flow-type=directional --source-zone=us-west --dest-zone=us-west-2

3. Create the pipe.

   Syntax

   radosgw-admin sync group pipe create --group-id=GROUP_ID --pipe-id=PIPE_ID  --source-zones='SOURCE_ZONE_NAME' --dest-zones='DESTINATION_ZONE_NAME'

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --group-id=group1 --pipe-id=pipe1 --source-zones='us-west' --dest-zones='us-west-2'

4. Update and commit the new period.

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

   Note

   Updating and committing the period is mandatory for a zonegroup policy.

5. Verify source and destination of zonegroup using sync info in both the sites.

   Syntax

   radosgw-admin sync info

Procedure

1. On the primary zone, create a new sync group with the status set to allowed.

   Syntax

   radosgw-admin sync group create --group-id=GROUP_ID --status=allowed --bucket=BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --group-id=group1 --status=allowed --bucket=buck

   Note

   Until a fully configured zonegroup replication policy is created, it is recommended to set the --status to allowed, to prevent the replication from starting.

2. Create the flow.

   Syntax

   radosgw-admin sync group flow create --bucket-name=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE_NAME --dest-zone=DESTINATION_ZONE_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync group flow create --bucket-name=buck --group-id=group1 --flow-id=us-west-backup --flow-type=directional --source-zone=us-west --dest-zone=us-west-2

3. Create the pipe.

   Syntax

   radosgw-admin sync group pipe create --group-id=GROUP_ID --bucket-name=BUCKET_NAME --pipe-id=PIPE_ID  --source-zones='SOURCE_ZONE_NAME' --dest-zones='DESTINATION_ZONE_NAME'

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --group-id=group1 --bucket-name=buck --pipe-id=pipe1 --source-zones='us-west' --dest-zones='us-west-2'

4. Verify source and destination of zonegroup using sync info in both the sites.

   Syntax

   radosgw-admin sync info --bucket-name=BUCKET_NAME

Procedure

1. Set the zonegroup policy --status to allowed to permit per-bucket replication.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id=group1 --status=allowed

2. Update the period after modifying the zonegroup policy.

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

3. Create a sync group for the bucket we want to synchronize to and set --status to enabled.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck \
                             --group-id=buck-default --status=enabled

4. Create a pipe for the group that was created in the previous step. The flow is inherited from the zonegroup level policy where data flow is symmetrical.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck \
                             --group-id=buck-default --pipe-id=pipe1 \
                             --source-zones='*' --dest-zones='*'

   Note

   Use wildcards * to specify the source and destination zones for the bucket replication.

5. Optional: To retrieve information about the expected bucket sync sources and targets, as defined by the sync policy, run the radosgw-admin bucket sync info command with the --bucket flag.

   Example

   [ceph: root@host01 /]# radosgw-admin bucket sync info --bucket buck
   realm 33157555-f387-44fc-b4b4-3f9c0b32cd66 (india)
   zonegroup 594f1f63-de6f-4e1e-90b6-105114d7ad55 (shared)
   zone ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5 (primary)
   bucket :buck[ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1]
   
   source zone e0e75beb-4e28-45ff-8d48-9710de06dcd0
   bucket :buck[ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1]

6. Optional: To retrieve information about the expected sync sources and targets, as defined by the sync policy, run the radosgw-admin sync info command with the --bucket flag.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket buck
   {
       "id": "pipe1",
       "source": {
           "zone": "secondary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "dest": {
           "zone": "primary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "params": {
           "source": {
               "filter": {
                   "tags": []
               }
           },
           "dest": {},
           "priority": 0,
           "mode": "system",
           "user": ""
       }
   },
       {
       "id": "pipe1",
       "source": {
           "zone": "primary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "dest": {
           "zone": "secondary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "params": {
           "source": {
               "filter": {
                   "tags": []
               }
           },
           "dest": {},
           "priority": 0,
           "mode": "system",
           "user": ""
       }
   }

1. Syncing from a bucket - You need to specify the source bucket.
2. Syncing to a bucket - You need to specify the destination bucket.

Syncing from a different bucket

1. Create a sync group to pull data from a bucket of another zone.

   Syntax

   radosgw-admin sync group create  --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck4 --group-id=buck4-default --status=enabled

2. Pull data.

   Syntax

   radosgw-admin sync group pipe create --bucket-name=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --source-zones=SOURCE_ZONE_NAME --source-bucket=SOURCE_BUCKET_NAME --dest-zones=DESTINATION_ZONE_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck4 \
                             --group-id=buck4-default --pipe-id=pipe1 \
                             --source-zones='*' --source-bucket=buck5 \
                             --dest-zones='*'

   In this example, you can see that the source bucket is buck5.

3. Optional: Sync from buckets in specific zones.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe modify --bucket=buck4 \
                             --group-id=buck4-default --pipe-id=pipe1 \
                             --source-zones=us-west --source-bucket=buck5 \
                             --dest-zones='*'

4. Check sync status.

   Syntax

   radosgw-admin sync info --bucket-name=BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket=buck4
   {
       "sources": [],
       "dests": [],
       "hints": {
           "sources": [],
           "dests": [
               "buck4:115b12b3-....14433.2"
           ]
       },
       "resolved-hints-1": {
           "sources": [],
           "dests": [
               {
                   "id": "pipe1",
                   "source": {
                       "zone": "us-west",
                       "bucket": "buck5"
                   },
                   "dest": {
                       "zone": "us-east",
                       "bucket": "buck4:115b12b3-....14433.2"
                   },
                   ...
               },
               {
                   "id": "pipe1",
                   "source": {
                       "zone": "us-west",
                       "bucket": "buck5"
                   },
                   "dest": {
                       "zone": "us-west-2",
                       "bucket": "buck4:115b12b3-....14433.2"
                   },
                   ...
               }
           ]
       },
       "resolved-hints": {
           "sources": [],
           "dests": []
       }

   Note that there are resolved-hints, which means that the bucket buck5 found about buck4 syncing from it indirectly, and not from its own policy. The policy for buck5 itself is empty.

Syncing to a different bucket

1. Create a sync group.

   Syntax

   radosgw-admin sync group create  --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck6 \
                             --group-id=buck6-default --status=enabled

2. Push data.

   Syntax

   radosgw-admin sync group pipe create --bucket-name=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --source-zones=SOURCE_ZONE_NAME --dest-zones=DESTINATION_ZONE_NAME --dest-bucket=DESTINATION_BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck6 \
                             --group-id=buck6-default --pipe-id=pipe1 \
                             --source-zones='*' --dest-zones='*' --dest-bucket=buck5

   In this example, you can see that the destination bucket is buck5.

3. Optional: Sync to buckets in specific zones.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe modify  --bucket=buck6 \
                             --group-id=buck6-default --pipe-id=pipe1 \
                             --source-zones='*' --dest-zones='us-west' --dest-bucket=buck5

4. Check sync status.

   Syntax

   radosgw-admin sync info --bucket-name=BUCKET_NAME

   Example

   [ceph: root@host01 /]#  radosgw-admin sync info --bucket buck5
   {
     "sources": [],
     "dests": [
       {
         "id": "pipe1",
         "source": {
           "zone": "us-west",
           "bucket": "buck6:c7887c5b-f6ff-4d5f-9736-aa5cdb4a15e8.20493.4"
         },
         "dest": {
           "zone": "us-east",
           "bucket": "buck5"
         },
         "params": {
           "source": {
             "filter": {
               "tags": []
             }
           },
           "dest": {},
           "priority": 0,
           "mode": "system",
           "user": "s3cmd"
         }
       },
     ],
     "hints": {
       "sources": [],
       "dests": [
         "buck5"
       ]
     },
     "resolved-hints-1": {
       "sources": [],
       "dests": []
     },
     "resolved-hints": {
       "sources": [],
       "dests": []
     }
   }

Procedure

1. Create a new sync group. If you want to create at the bucket level, use the --bucket option.

   Syntax

   radosgw-admin sync group create --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck1 --group-id=buck8-default --status=enabled

2. Sync between buckets where the object matches the tags. The flow is inherited from the zonegroup level policy where data flow is symmetrical.

   Syntax

   radosgw-admin sync group pipe create --bucket=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --tags-add=KEY1=VALUE1,KEY2=VALUE2 --source-zones='ZONE_NAME1','ZONE_NAME2' --dest-zones='ZONE_NAME1','ZONE_NAME2'

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck1 \
                             --group-id=buck1-default --pipe-id=pipe-tags \
                             --tags-add=color=blue,color=red --source-zones='*' \
                             --dest-zones='*'

3. Sync between buckets where the object matches the prefix. The flow is inherited from the zonegroup level policy where data flow is symmetrical.

   Syntax

   radosgw-admin sync group pipe create --bucket=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --prefix=PREFIX --source-zones='ZONE_NAME1','ZONE_NAME2' --dest-zones='ZONE_NAME1','ZONE_NAME2'

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck1 \
                             --group-id=buck1-default --pipe-id=pipe-prefix \
                             --prefix=foo/ --source-zones='*' --dest-zones='*' \

4. Check the updated sync.

   Syntax

   radosgw-admin sync info --bucket=BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket=buck1

   Note

   In the example, only two different destinations and no sources reflect, one each for configuration. When the sync process happens, it selects the relevant rule for each object it syncs.

Procedure

1. Run the sync group modify command to change the status from allowed to forbidden.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id  buck-default --status forbidden --bucket buck
   {
       "groups": [
           {
               "id": "buck-default",
               "data_flow": {},
               "pipes": [
                   {
                       "id": "pipe1",
                       "source": {
                           "bucket": "*",
                           "zones": [
                               "*"
                           ]
                       },
                       "dest": {
                           "bucket": "*",
                           "zones": [
                               "*"
                           ]
                       },
                       "params": {
                           "source": {
                               "filter": {
                                   "tags": []
                               }
                           },
                           "dest": {},
                           "priority": 0,
                           "mode": "system",
                       }
                   }
               ],
               "status": "forbidden"
           }
       ]
   }

   In this example, the replication of the bucket buck is interrupted between zones us-east and us-west.

   Note

   No update and commit for the period is required as this is a bucket sync policy.

2. Optional: Run sync info command command to check the status of the sync for bucket buck.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket buck
   {
       "sources": [],
       "dests": [],
       "hints": {
           "sources": [],
           "dests": []
       },
       "resolved-hints-1": {
           "sources": [],
           "dests": []
       },
       "resolved-hints": {
           "sources": [],
           "dests": []
       }
   }

   Note

   There are no source and destination targets as the replication is interrupted.