Object Gateway Guide for Red Hat Enterprise Linux

A radosgw user needs to be created and granted access. The command man radosgw-admin will provide information on additional command options.

To create the user, execute the following on the gateway host:

sudo radosgw-admin user create --uid="testuser" --display-name="First User"

The output of the command will be something like the following:

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [],
	"keys": [{
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

A Swift subuser needs to be created if this kind of access is needed. Creating a Swift user is a two step process. The first step is to create the user. The second is to create the secret key.

Execute the following steps on the gateway host:

Create the Swift user:

sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full

The output will be something like the following:

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [{
		"id": "testuser:swift",
		"permissions": "full-control"
	}],
	"keys": [{
		"user": "testuser:swift",
		"access_key": "3Y1LNW4Q6X0Y53A52DET",
		"secret_key": ""
	}, {
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

Create the secret key:

sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret

The output will be something like the following:

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [{
		"id": "testuser:swift",
		"permissions": "full-control"
	}],
	"keys": [{
		"user": "testuser:swift",
		"access_key": "3Y1LNW4Q6X0Y53A52DET",
		"secret_key": ""
	}, {
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [{
		"user": "testuser:swift",
		"secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA"
	}],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

You need to write and run a Python test script for verifying S3 access. The S3 access test script will connect to the radosgw, create a new bucket and list all buckets. The values for aws_access_key_id and aws_secret_access_key are taken from the values of access_key and secret_key returned by the radosgw_admin command.

Execute the following steps:

Swift access can be verified via the swift command line client. The command man swift will provide more information on available command line options.

To install swift client, execute the following:

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

To test swift access, execute the following:

swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list

Replace {IP ADDRESS} with the public IP address of the gateway server and {swift_secret_key} with its value from the output of radosgw-admin key create command executed for the swift user. Replace {port} with the port number you are using with Civetweb (e.g., 7480 is the default). If you don’t replace the port, it will default to port 80.

For example:

swift -A http://10.19.143.116:7480/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list

The output should be:

my-new-bucket

To set up an HA Proxy with the Ceph Object Gateway, you must have:

For a detailed discussion of HAProxy and keepalived, see Load Balancer Administration.

The following setup assumes two HAProxy nodes named haproxy and haproxy2, and two Ceph Object Gateway servers named rgw1 and rgw2. You may use any naming convention you prefer. Perform the following procedure on your at least two HAProxy nodes:

Perform the following procedure on your at least two HAProxy nodes:

Perform the following procedure on your at least two HAProxy nodes:

On your HAProxy nodes, check to ensure the virtual IP address from your keepalived configuration appears.

ip addr show

On your calamari node, see if you can reach the gateway nodes via the load balancer configuration. For example:

wget haproxy

This should return the same result as:

wget rgw1

If it returns an index.html file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
	<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
		<Owner>
			<ID>anonymous</ID>
			<DisplayName></DisplayName>
		</Owner>
		<Buckets>
		</Buckets>
	</ListAllMyBucketsResult>

Then, your configuration is working properly.

Static web hosting requires at least one running Ceph Storage Cluster, and at least two Ceph Object Gateway instances for static web sites. Red Hat assumes that each zone will have multiple gateway instances load balanced by HAProxy/keepalived.

See HAProxy/keepalived Configuration for additional details on HAProxy/keepalived.

Static web hosting functionality uses its own API, so configuring a gateway to use static web sites in S3 buckets requires the following:

To enable a gateway for static web hosting, edit the Ceph configuration file and add the following settings:

[client.rgw.<STATIC-SITE-HOSTNAME>]
...
rgw_enable_static_website = true
rgw_enable_apis = s3website
rgw_dns_name = objects-zonegroup.domain.com
rgw_dns_s3website_name = objects-website-zonegroup.domain.com
rgw_resolve_cname = true
...

The rgw_enable_static_website setting MUST be true. The rgw_enable_apis setting MUST enable the s3website API. The rgw_dns_name and rgw_dns_s3website_name settings must provide their fully qualified domains. If the site will use canonical name extensions, set rgw_resolve_cname to true.

The following is an example of assumed DNS settings, where the first two lines specify the domains of the the gateway instance using a standard S3 interface and point to the IPv4 and IPv6 addresses respectively. The third line provides a wildcard CNAME setting for S3 buckets using canonical name extensions. The fourth and fifth lines specify the domains for the gateway instance using the S3 website interface and point to their IPv4 and IPv6 addresses respectively.

objects-zonegroup.domain.com. IN    A 192.0.2.10
objects-zonegroup.domain.com. IN AAAA 2001:DB8::192:0:2:10
*.objects-zonegroup.domain.com. IN CNAME objects-zonegroup.domain.com.
objects-website-zonegroup.domain.com. IN    A 192.0.2.20
objects-website-zonegroup.domain.com. IN AAAA 2001:DB8::192:0:2:20

If using Ceph Object Gateway in a multi-site configuration, consider using a routing solution to route traffic to the gateway closest to the client.

The Amazon Web Service (AWS) requires static web host buckets to match the host name. Ceph provides a few different ways to configure the DNS, and HTTPS will work if the proxy has a matching certificate.

*.objects-website-zonegroup.domain.com. IN CNAME objects-website-zonegroup.domain.com.

Access the bucket name in the following manner:

http://bucket1.objects-website-zonegroup.domain.com

Where the bucket name is bucket1.

www.example.com. IN CNAME bucket2.objects-website-zonegroup.domain.com.

Where the bucket name is bucket2.

Access the bucket in the following manner:

http://www.example.com

www.example.com. IN CNAME www.example.com.objects-website-zonegroup.domain.com.

Access the bucket in the following manner:

http://www.example.com

www.example.com. IN A 192.0.2.20
www.example.com. IN AAAA 2001:DB8::192:0:2:20

Access the bucket in the following manner:

http://www.example.com

To create a static website perform the following steps:

In Red Hat Ceph Storage 2, the Ceph Object Gateway supports multi-tenancy for both the S3 and Swift APIs, where each user and bucket lies under a "tenant." Multi tenancy prevents namespace clashing when multiple tenants are using common bucket names, such as "test", "main" and so forth.

Each user and bucket lies under a tenant. For backward compatibility, a "legacy" tenant with an empty name is added. Whenever referring to a bucket without specifically specifying a tenant, the Swift API will assume the "legacy" tenant. Existing users are also stored under the legacy tenant, so they will access buckets and objects the same way as earlier releases.

Tenants as such do not have any operations on them. They appear and and disappear as needed, when users are administered. In order to create, modify, and remove users with explicit tenants, either an additional option --tenant is supplied, or a syntax "<tenant>$<user>" is used in the parameters of the radosgw-admin command.

To create a user testx$tester for S3, execute the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester --display-name "Test User" --access_key TESTER --secret test123 user create

To create a user testx$tester for Swift, execute one of the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester --display-name "Test User" --subuser tester:test --key-type swift --access full user create

[root@master-zone]# radosgw-admin key create --subuser 'testx$tester:test' --key-type swift --secret test123

Use the user create command to create an S3-interface user. You MUST specify a user ID and a display name. You may also specify an email address. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

[root@master-zone]# radosgw-admin user create --uid=<id> \
[--key-type=<type>] [--gen-access-key|--access-key=<key>]\
[--gen-secret | --secret=<key>] \
[--email=<email>] --display-name=<name>

For example:

[root@master-zone]# radosgw-admin user create --uid=janedoe --display-name="Jane Doe" --email=jane@example.com

{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

To create a subuser (Swift interface), you must specify the user ID (--uid={username}), a subuser ID and the access level for the subuser. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

[root@master-zone]# radosgw-admin subuser create --uid={uid} --subuser={uid} --access=[ read | write | readwrite | full ]

For example:

[root@master-zone]# radosgw-admin subuser create --uid=janedoe --subuser=janedoe:swift --access=full

{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "janedoe:swift",
          "permissions": "full-control"}],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

To get information about a user, you must specify user info and the user ID (--uid={username}) . :

radosgw-admin user info --uid=janedoe

To modify information about a user, you must specify the user ID (--uid={username}) and the attributes you want to modify. Typical modifications are to keys and secrets, email addresses, display names and access levels. For example:

[root@master-zone]# radosgw-admin user modify --uid=janedoe --display-name="Jane E. Doe"

To modify subuser values, specify subuser modify and the subuser ID. For example:

[root@master-zone]# radosgw-admin subuser modify --subuser=janedoe:swift --access=full

When you create a user, the user is enabled by default. However, you may suspend user privileges and re-enable them at a later time. To suspend a user, specify user suspend and the user ID. :

[root@master-zone]# radosgw-admin user suspend --uid=johndoe

To re-enable a suspended user, specify user enable and the user ID. :

[root@master-zone]# radosgw-admin user enable --uid=johndoe

When you remove a user, the user and subuser are removed from the system. However, you may remove just the subuser if you wish. To remove a user (and subuser), specify user rm and the user ID.

[root@master-zone]# radosgw-admin user rm --uid=<uid> [--purge-keys] [--purge-data]

For example:

[root@master-zone]# radosgw-admin user rm --uid=johndoe --purge-data

To remove the subuser only, specify subuser rm and the subuser name.

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:swift --purge-keys

Options include:

When you remove a sub user, you are removing access to the Swift interface. The user will remain in the system. The Ceph Object Gateway To remove the subuser, specify subuser rm and the subuser ID. :

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:test

Options include:

To create a key for a user, you must specify key create. For a user, specify the user ID and the s3 key type. To create a key for subuser, you must specify the subuser ID and the swift keytype. For example:

[root@master-zone]# radosgw-admin key create --subuser=johndoe:swift --key-type=swift --gen-secret

{ "user_id": "johndoe",
  "rados_uid": 0,
  "display_name": "John Doe",
  "email": "john@example.com",
  "suspended": 0,
  "subusers": [
     { "id": "johndoe:swift",
       "permissions": "full-control"}],
  "keys": [
    { "user": "johndoe",
      "access_key": "QFAMEDSJP5DEKJO0DDXY",
      "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
  "swift_keys": [
    { "user": "johndoe:swift",
      "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}

Users and subusers must have access keys to use the S3 and Swift interfaces. When you create a user or subuser and you do not specify an access key and secret, the key and secret get generated automatically. You may create a key and either specify or generate the access key and/or secret. You may also remove an access key and secret. Options include:

To add a key, specify the user. :

[root@master-zone]# radosgw-admin key create --uid=johndoe --key-type=s3 --gen-access-key --gen-secret

You may also specify a key and a secret.

To remove an access key, specify the user. :

[root@master-zone]# radosgw-admin key rm --uid=johndoe

The Ceph Storage Cluster provides an administrative API that enables users to execute administrative functions via the REST API. By default, users DO NOT have access to this API. To enable a user to exercise administrative functionality, provide the user with administrative capabilities.

To add administrative capabilities to a user, execute the following:

[root@master-zone]# radosgw-admin caps add --uid={uid} --caps={caps}

You can add read, write or all capabilities to users, buckets, metadata and usage (utilization). For example:

--caps="[users|buckets|metadata|usage|zone]=[*|read|write|read, write]"

For example:

[root@master-zone]# radosgw-admin caps add --uid=johndoe --caps="users=*"

To remove administrative capabilities from a user, execute the following:

[root@master-zone]# radosgw-admin caps remove --uid=johndoe --caps={caps}

Before you enable a quota, you must first set the quota parameters. For example:

[root@master-zone]# radosgw-admin quota set --quota-scope=user --uid=<uid> [--max-objects=<num objects>] [--max-size=<max size>]

For example:

[root@master-zone]# radosgw-admin quota set --quota-scope=user --uid=johndoe --max-objects=1024 --max-size=1024

A negative value for num objects and / or max size means that the specific quota attribute check is disabled.

Once you set a user quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=user --uid=<uid>

You may disable an enabled user quota. For example:

[root@master-zone]# radosgw-admin quota disable --quota-scope=user --uid=<uid>

Bucket quotas apply to the buckets owned by the specified uid. They are independent of the user. :

[root@master-zone]# radosgw-admin quota set --uid=<uid> --quota-scope=bucket [--max-objects=<num objects>] [--max-size=<max size]

A negative value for num objects and / or max size means that the specific quota attribute check is disabled.

Once you set a bucket quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=bucket --uid=<uid>

You may disable an enabled bucket quota. For example:

[root@master-zone]# radosgw-admin quota-disable --quota-scope=bucket --uid=<uid>

You may access each user’s quota settings via the user information API. To read user quota setting information with the CLI interface, execute the following:

[root@master-zone]# radosgw-admin user info --uid=<uid>

Quota stats get updated asynchronously. You can update quota statistics for all users and all buckets manually to retrieve the latest quota stats. :

[root@master-zone]# radosgw-admin user stats --uid=<uid> --sync-stats

To see how much of the quota a user has consumed, execute the following:

[root@master-zone]# radosgw-admin user stats --uid=<uid>

Quota statistics are cached for each Ceph Gateway instance. If there are multiple instances, then the cache can keep quotas from being perfectly enforced, as each instance will have a different view of the quotas. The options that control this are rgw bucket quota ttl, rgw user quota bucket sync interval and rgw user quota sync interval. The higher these values are, the more efficient quota operations are, but the more out-of-sync multiple instances will be. The lower these values are, the closer to perfect enforcement multiple instances will achieve. If all three are 0, then quota caching is effectively disabled, and multiple instances will have perfect quota enforcement. See Chapter 4, Object Gateway Configuration Reference for more details on these options.

You can read and write quota settings in a region map. To get a region map:

[root@master-zone]# radosgw-admin regionmap get > regionmap.json

To set quota settings for the entire region, modify the quota settings in the region map. Then, use the regionmap set command to update the region map:

[root@master-zone]# radosgw-admin regionmap set < regionmap.json

To show usage statistics, specify the usage show. To show usage for a particular user, you must specify a user ID. You may also specify a start date, end date, and whether or not to show log entries.:

radosgw-admin usage show --uid=johndoe --start-date=2012-03-01 --end-date=2012-04-01

You may also show a summary of usage information for all users by omitting a user ID. :

radosgw-admin usage show --show-log-entries=false

With heavy use, usage logs can begin to take up storage space. You can trim usage logs for all users and for specific users. You may also specify date ranges for trim operations. :

[root@master-zone]# radosgw-admin usage trim --start-date=2010-01-01 --end-date=2010-12-31
[root@master-zone]# radosgw-admin usage trim --uid=johndoe
[root@master-zone]# radosgw-admin usage trim --uid=johndoe --end-date=2013-12-31

Normally, in a healthy storage cluster you should not have any leaking objects, but in some cases leaky objects can occur. For example, if the RADOS Gateway goes down in the middle of an operation, this may cause some RADOS objects to become orphans. Also, unknown bugs may cause these orphan objects to occur. The radosgw-admin command provides you a tool to search for these orphan objects and clean them up. With the --pool option, you can specify which pool to scan for leaky RADOS objects. With the --num-shards option, you may specify the number of shards to use for keeping temporary scan data.

To create a realm, execute realm create and specify the realm name. If the realm is the default, specify --default.

[root@master-zone]# radosgw-admin realm create --rgw-realm={realm-name} [--default]

For example:

[root@master-zone]# radosgw-admin realm create --rgw-realm=movies --default

By specifying --default, the realm will be called implicitly with each radosgw-admin call unless --rgw-realm and the realm name are explicitly provided.

One realm in the list of realms should be the default realm. There may be only one default realm. If there is only one realm and it wasn’t specified as the default realm when it was created, make it the default realm. Alternatively, to change which realm is the default, execute:

[root@master-zone]# radosgw-admin realm default --rgw-realm=movies

To delete a realm, execute realm delete and specify the realm name.

[root@master-zone]# radosgw-admin realm delete --rgw-realm={realm-name}

For example:

[root@master-zone]# radosgw-admin realm delete --rgw-realm=movies

To get a realm, execute realm get and specify the realm name.

#radosgw-admin realm get --rgw-realm=<name>

For example:

# radosgw-admin realm get --rgw-realm=movies [> filename.json]

The CLI will echo a JSON object with the realm properties.

{
    "id": "0a68d52e-a19c-4e8e-b012-a8f831cb3ebc",
    "name": "movies",
    "current_period": "b0c5bbef-4337-4edd-8184-5aeab2ec413b",
    "epoch": 1
}

Use > and an output file name to output the JSON object to a file.

To set a realm, execute realm set, specify the realm name, and --infile= with an input file name.

[root@master-zone]# radosgw-admin realm set --rgw-realm=<name> --infile=<infilename>

For example:

[root@master-zone]# radosgw-admin realm set --rgw-realm=movies --infile=filename.json

To list realms, execute realm list.

# radosgw-admin realm list

To list realm periods, execute realm list-periods.

# radosgw-admin realm list-periods

To pull a realm from the node containing the master zone group and master zone to a node containing a secondary zone group or zone, execute realm pull on the node that will receive the realm configuration.

# radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}

A realm is not part of the period. Consequently, renaming the realm is only applied locally, and will not get pulled with realm pull. When renaming a realm with multiple zones, run the command on each zone. To rename a realm, execute the following:

# radosgw-admin realm rename --rgw-realm=<current-name> --realm-new-name=<new-realm-name>

Creating a zone group consists of specifying the zone group name. Creating a zone assumes it will live in the default realm unless --rgw-realm=<realm-name> is specified. If the zonegroup is the default zonegroup, specify the --default flag. If the zonegroup is the master zonegroup, specify the --master flag. For example:

# radosgw-admin zonegroup create --rgw-zonegroup=<name> [--rgw-realm=<name>][--master] [--default]

One zonegroup in the list of zonegroups should be the default zonegroup. There may be only one default zonegroup. If there is only one zonegroup and it wasn’t specified as the default zonegroup when it was created, make it the default zonegroup. Alternatively, to change which zonegroup is the default, execute:

# radosgw-admin zonegroup default --rgw-zonegroup=comedy

Then, update the period:

# radosgw-admin period update --commit

To add a zone to a zonegroup, execute the following:

# radosgw-admin zonegroup add --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit

To remove a zone from a zonegroup, execute the following:

# radosgw-admin zonegroup remove --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit

To rename a zonegroup, execute the following:

# radosgw-admin zonegroup rename --rgw-zonegroup=<name> --zonegroup-new-name=<name>

Then, update the period:

# radosgw-admin period update --commit

To delete a zonegroup, execute the following:

# radosgw-admin zonegroup delete --rgw-zonegroup=<name>

Then, update the period:

# radosgw-admin period update --commit

A Ceph cluster contains a list of zone groups. To list the zone groups, execute:

# radosgw-admin zonegroup list

The radosgw-admin returns a JSON formatted list of zone groups.

{
    "default_info": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "zonegroups": [
        "us"
    ]
}

To list the details of each zone grouop, execute:

# radosgw-admin zonegroup-map get

To view the configuration of a zone group, execute:

# radosgw-admin zonegroup get [--rgw-zonegroup=<zonegroup>]

The zone group configuration looks like this:

{
    "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "name": "us",
    "api_name": "us",
    "is_master": "true",
    "endpoints": [
        "http:\/\/rgw1:80"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
    "zones": [
        {
            "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
            "name": "us-east",
            "endpoints": [
                "http:\/\/rgw1"
            ],
            "log_meta": "true",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        },
        {
            "id": "d1024e59-7d28-49d1-8222-af101965a939",
            "name": "us-west",
            "endpoints": [
                "http:\/\/rgw2:80"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
}

Defining a zone group consists of creating a JSON object, specifying at least the required settings:

To set a zone group, create a JSON object consisting of the required fields, save the object to a file (e.g., zonegroup.json); then, execute the following command:

# radosgw-admin zonegroup set --infile zonegroup.json

Where zonegroup.json is the JSON file you created.

Finally, update the period:

# radosgw-admin period update --commit

Setting a zone group map consists of creating a JSON object consisting of one or more zone groups, and setting the master_zonegroup for the cluster. Each zone group in the zone group map consists of a key/value pair, where the key setting is equivalent to the name setting for an individual zone group configuration, and the val is a JSON object consisting of an individual zone group configuration.

You may only have one zone group with is_master equal to true, and it must be specified as the master_zonegroup at the end of the zone group map. The following JSON object is an example of a default zone group map.

{
    "zonegroups": [
        {
            "key": "90b28698-e7c3-462c-a42d-4aa780d24eda",
            "val": {
                "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http:\/\/rgw1:80"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
                "zones": [
                    {
                        "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw1"
                        ],
                        "log_meta": "true",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    },
                    {
                        "id": "d1024e59-7d28-49d1-8222-af101965a939",
                        "name": "us-west",
                        "endpoints": [
                            "http:\/\/rgw2:80"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
            }
        }
    ],
    "master_zonegroup": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    }
}

To set a zone group map, execute the following:

# radosgw-admin zonegroup-map set --infile zonegroupmap.json

Where zonegroupmap.json is the JSON file you created. Ensure that you have zones created for the ones specified in the zone group map. Finally, update the period.

# radosgw-admin period update --commit

To create a zone, specify a zone name. If it is a master zone, specify the --master option. Only one zone in a zone group may be a master zone. To add the zone to a zonegroup, specify the --rgw-zonegroup option with the zonegroup name.

[root@new-zone]# radosgw-admin zone create --rgw-zone=<name> \
                [--zonegroup=<zonegroup-name]\
                [--endpoints=<endpoint>[,<endpoint>] \
                [--master] [--default] \
                --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY

Then, update the period:

# radosgw-admin period update --commit

To delete zone, first remove it from the zonegroup.

# radosgw-admin zonegroup remove --rgw-zonegroup=<name>\
                                 --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit

Next, delete the zone.

Execute the following:

[root@zone]# radosgw-admin zone delete --rgw-zone<name>

Finally, update the period:

# radosgw-admin period update --commit

If the pools for the deleted zone will not be used anywhere else, consider deleting the pools. Replace <del-zone> in the example below with the deleted zone’s name.

# rados rmpool <del-zone>.rgw.control <del-zone>.rgw.control --yes-i-really-really-mean-it
# rados rmpool <del-zone>.rgw.data.root <del-zone>.rgw.data.root --yes-i-really-really-mean-it
# rados rmpool <del-zone>.rgw.gc <del-zone>.rgw.gc --yes-i-really-really-mean-it
# rados rmpool <del-zone>.rgw.log <del-zone>.rgw.log --yes-i-really-really-mean-it
# rados rmpool <del-zone>.rgw.users.uid <del-zone>.rgw.users.uid --yes-i-really-really-mean-it

To modify a zone, specify the zone name and the parameters you wish to modify.

[root@zone]# radosgw-admin zone modify [options]

--access-key=<key>--secret/--secret-key=<key>--master--default--endpoints=<list>

Then, update the period:

# radosgw-admin period update --commit

As root, to list the zones in a cluster, execute:

# radosgw-admin zone list

As root, to get the configuration of a zone, execute:

# radosgw-admin zone get [--rgw-zone=<zone>]

The default zone looks like this:

{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "", "secret_key": ""},
  "placement_pools": [
      {  "key": "default-placement",
         "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets"}
      }
    ]
  }

Configuring a zone involves specifying a series of Ceph Object Gateway pools. For consistency, we recommend using a pool prefix that is the same as the zone name.

To set a zone, create a JSON object consisting of the pools, save the object to a file; then, execute the following command on a node within the zone, replacing {zone-name} with the name of the zone:

[root@zone]# radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json

Where zone.json is the JSON file you created.

Then, as root, update the period:

# radosgw-admin period update --commit

To rename a zone, specify the zone name and the new zone name.

[root@zone]# radosgw-admin zone rename --rgw-zone=<name> --zone-new-name=<name>

Then, update the period:

# radosgw-admin period update --commit

The procedure for installing Ceph Object Gateway is identical to the procedure described in the Red Hat Ceph Storage Installation Guide for Red Hat Enterprise Linux but with this difference: proceed to configure the zones BEFORE starting the Ceph Object Gateway daemons to avoid the need to remove default zones and pools.

Ansible can install and configure Ceph Object Gateways for use with a Ceph Storage cluster. So for a simple configuration, add Ceph Object Gateways to your Ansible configuration. For multi-site and multi-site group deployments, you should have an Ansible configuration for each zone.

If you install Ceph Object Gateway with Ansible, the Ansible playbooks will handle the initial configuration for you. To install the Ceph Object Gateway with Ansible, add your hosts to the /etc/ansible/hosts file. Add the Ceph Object Gateway hosts under an [rgws] section to identify their roles to Ansible. If your hosts have sequential naming, you may use a range. For example:

[rgws]
<rgw-host-name-1>
<rgw-host-name-2>
<rgw-host-name[3..10]>

Once you have added the hosts, you may rerun your Ansible playbooks.

When updating an existing multi-site cluster with an asynchronous update, follow the installation instruction for the update. Then, restart the gateway instances.

A realm contains the multi-site configuration of zone groups and zones and also serves to enforce a globally unique namespace within the realm.

Create a new realm for the multi-site configuration by opening a command line interface on a host identified to serve in the master zone group and zone. Then, execute the following:

[root@master-zone]# radosgw-admin realm create --rgw-realm={realm-name} [--default]

For example:

[root@master-zone]# radosgw-admin realm create --rgw-realm=movies --default

If the cluster will have a single realm, specify the --default flag. If --default is specified, radosgw-admin will use this realm by default. If --default is not specified, adding zone-groups and zones requires specifying either the --rgw-realm flag or the --realm-id flag to identify the realm when adding zone groups and zones.

After creating the realm, radosgw-admin will echo back the realm configuration. For example:

{
    "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62",
    "name": "movies",
    "current_period": "1950b710-3e63-4c41-a19e-46a715000980",
    "epoch": 1
}

A realm must have at least one zone group, which will serve as the master zone group for the realm.

Create a new master zone group for the multi-site configuration by opening a command line interface on a host identified to serve in the master zone group and zone. Then, execute the following:

[root@master-zone]# radosgw-admin zonegroup create --rgw-zonegroup={name} --endpoints={url} [--rgw-realm={realm-name}|--realm-id={realm-id}] --master --default

For example:

[root@master-zone]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=movies --master --default

If the realm will only have a single zone group, specify the --default flag. If --default is specified, radosgw-admin will use this zone group by default when adding new zones. If --default is not specified, adding zones will require either the --rgw-zonegroup flag or the --zonegroup-id flag to identify the zone group when adding or modifying zones.

After creating the master zone group, radosgw-admin will echo back the zone group configuration. For example:

{
    "id": "f1a233f5-c354-4107-b36c-df66126475a6",
    "name": "us",
    "api_name": "us",
    "is_master": "true",
    "endpoints": [
        "http:\/\/rgw1:80"
    ],
    "hostnames": [],
    "hostnames_s3webzone": [],
    "master_zone": "",
    "zones": [],
    "placement_targets": [],
    "default_placement": "",
    "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62"
}

Create a new master zone for the multi-site configuration by opening a command line interface on a host identified to serve in the master zone group and zone. Then, execute the following:

[root@master-zone]# radosgw-admin zone create \
                            --rgw-zonegroup={zone-group-name} \
                            --rgw-zone={zone-name} \
                            --master --default \
                            --endpoints={http://fqdn}[,{http://fqdn}

For example:

[root@master-zone]# radosgw-admin zone create --rgw-zonegroup=us \
                            --rgw-zone=us-east \
                            --master --default \
                            --endpoints={http://fqdn}[,{http://fqdn}

Delete the default zone if it exists. Make sure to remove it from the default zone group first.

[root@master-zone]# radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
[root@master-zone]# radosgw-admin period update --commit
[root@master-zone]# radosgw-admin zone delete --rgw-zone=default
[root@master-zone]# radosgw-admin period update --commit
[root@master-zone]# radosgw-admin zonegroup delete --rgw-zonegroup=default
[root@master-zone]# radosgw-admin period update --commit

Finally, delete the default pools in your Ceph storage cluster if they exist.

# rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it
# rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
# rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
# rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
# rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it

The ceph-radosgw daemons must authenticate before pulling realm and period information. In the master zone, create a system user to facilitate authentication between daemons.

[root@master-zone]# radosgw-admin user create --uid="{user-name}" --display-name="{Display Name}" --system

For example:

[root@master-zone]# radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system

Make a note of the access_key and secret_key, as the secondary zones will require them to authenticate with the master zone.

Finally, add the system user to the master zone.

[root@master-zone]# radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key} --secret={secret}
[root@master-zone]# radosgw-admin period update --commit

After updating the master zone configuration, update the period.

# radosgw-admin period update --commit

Update the Ceph configuration file on master zone hosts by adding the rgw_zone configuration option and the name of the master zone to the instance entry.

[client.rgw.{instance-name}]
...
rgw_zone={zone-name}

For example:

[client.rgw.rgw1]
host = rgw1
rgw frontends = "civetweb port=80"
rgw_zone=us-east

On the object gateway host, start and enable the Ceph Object Gateway service:

# systemctl start ceph-radosgw@rgw.`hostname -s`
# systemctl enable ceph-radosgw@rgw.`hostname -s`

If the service is already running, restart the service instead of starting and enabling it:

# systemctl restart ceph-radosgw@rgw.`hostname -s`

Using the URL path, access key and secret of the master zone in the master zone group, pull the realm to the host. To pull a non-default realm, specify the realm using the --rgw-realm or --realm-id configuration options.

# radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}

If this realm is the default realm or the only realm, make the realm the default realm.

# radosgw-admin realm default --rgw-realm={realm-name}

Using the URL path, access key and secret of the master zone in the master zone group, pull the period to the host. To pull a period from a non-default realm, specify the realm using the --rgw-realm or --realm-id configuration options.

# radosgw-admin period pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}

Create a secondary zone for the multi-site configuration by opening a command line interface on a host identified to serve the secondary zone. Specify the zone group ID, the new zone name and an endpoint for the zone. DO NOT use the --master or --default flags. In Red Hat Ceph Storage 2, all zones run in an active-active configuration by default; that is, a gateway client may write data to any zone and the zone will replicate the data to all other zones within the zone group. If the secondary zone should not accept write operations, specify the --read-only flag to create an active-passive configuration between the master zone and the secondary zone. Additionally, provide the access_key and secret_key of the generated system user stored in the master zone of the master zone group. Execute the following:

[root@second-zone]# radosgw-admin zone create --rgw-zonegroup={zone-group-name}\
                           --rgw-zone={zone-name} --endpoints={url} \
                           --access-key={system-key} --secret={secret}\
                           --endpoints=http://{fqdn}:80 \
                           [--read-only]

For example:

[root@second-zone]# radosgw-admin zone create --rgw-zonegroup=us \
                            --rgw-zone=us-west \
                            --access-key={system-key} --secret={secret} \
                            --endpoints=http://rgw2:80

Delete the default zone if needed.

[root@second-zone]# radosgw-admin zone delete --rgw-zone=default

Finally, delete the default pools in your Ceph storage cluster if needed.

# rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it
# rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
# rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
# rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
# rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it

Update the Ceph configuration file on the secondary zone hosts by adding the rgw_zone configuration option and the name of the secondary zone to the instance entry.

[client.rgw.{instance-name}]
...
rgw_zone={zone-name}

For example:

[client.rgw.rgw2]
host = rgw2
rgw frontends = "civetweb port=80"
rgw_zone=us-west

After updating the master zone configuration, update the period.

# radosgw-admin period update --commit

On the object gateway host, start and enable the Ceph Object Gateway service:

# systemctl start ceph-radosgw@rgw.`hostname -s`
# systemctl enable ceph-radosgw@rgw.`hostname -s`

If the service is already running, restart the service instead of starting and enabling it:

# systemctl restart ceph-radosgw@rgw.`hostname -s`