Chapter 9. Federated Object Gateway

Before configuring the cluster, defining region, zone and instance names will help you manage your cluster. Let’s assume the region represents the United States, and we refer to it by its standard abbreviation.

Let’s assume the zones represent the Eastern and Western United States. For continuity, our naming convention will use {region name}-{zone name} format, but you can use any naming convention you prefer.

Finally, let’s assume that zones may have more than one Ceph Object Gateway instance per zone. For continuity, our naming convention will use {region name}-{zone name}-{instance} format, but you can use any naming convention you prefer.

For Red Hat Ceph Storage v1.3, Red Hat supports the Ceph Object Gateway running on Civetweb (embedded into the ceph-radosgw daemon) instead of Apache and FastCGI. Using Civetweb simplifies the installation and configuration.

We will configure one Ceph Object Gateway instance per node and for ease of understanding, we will use the Ceph Object Gateway instance names us-east-1, us-west-1 mentioned in Naming for the Master Region as the short hostnames (hostname -s) for Ceph Object Gateway nodes of master region and the Ceph Object Gateway instance names eu-east-1 and eu-west-1 mentioned in Naming for the Secondary Region as the short hostnames (hostname -s) for Ceph Object Gateway nodes of secondary region.

Refer to the Red Hat Ceph Storage Installation Guide for RHEL, and execute the pre-installation procedures on your Ceph Object Gateway nodes of master region i.e, us-east-1 and us-west-1. Specifically, you should enable password-less ssh to these nodes from admin node, disable requiretty, set SELinux to Permissive and set up a Ceph Deploy user with password-less sudo access on these nodes. For Ceph Object Gateways, you will need to open the port that Civetweb will use in production.

Red Hat packages the Ceph Object Gateway in the rhel-7-server-rhceph-1.3-tools-rpms repository. To ensure you are using the same version of Ceph as your storage cluster, execute the following to enable the repository on your Ceph Object Gateway nodes of master region i.e, us-east-1 and us-west-1:

sudo subscription-manager repos --enable=rhel-7-server-rhceph-1.3-tools-rpms

From the working directory of your admin node, install the Ceph Object Gateway package on your Ceph Object Gateway nodes of master region i.e, us-east-1 and us-west-1:

ceph-deploy install --rgw us-east-1 us-west-1

The ceph-common package is a dependency, so ceph-deploy will install this too along with ceph-radosgw package. The ceph CLI tools are intended for administrators. To make your Ceph Object Gateway nodes administrator nodes, execute the following from the working directory of your admin node (e.g. ceph-config):

ceph-deploy admin us-east-1 us-west-1

The Ceph Object Gateway synchronization agent radosgw-agent is required for data synchronization between zones within a region and metadata synchronization between two regions.

To install the Ceph Object Gateway synchronization agent, execute the following on your Ceph Object Gateway nodes of master region i.e, us-east-1 and us-west-1:

sudo yum install radosgw-agent

From the working directory of your admin node, create instances of Ceph Object Gateway on your Ceph Object Gateway nodes of master region i.e, us-east-1 and us-west-1:

ceph-deploy rgw create us-east-1 us-west-1

The above command creates gateway username, keyring, data directory for each Ceph Object Gateway node and places the keyring in the newly created data directory /var/lib/ceph/radosgw/{rgw-intance}. The command also provides write capability to the keyring so that each instance can create pools automatically.

Once a gateway instance is running, you should be able to access it on port 7480 with an unauthenticated request like this:

http://us-east-1:7480

If the gateway instance is working properly, you should receive a response like this:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Owner>
		<ID>anonymous</ID>
		<DisplayName></DisplayName>
	</Owner>
	<Buckets>
	</Buckets>
</ListAllMyBucketsResult>

If at any point you run into trouble and you want to start over, execute the following to purge the configuration:

ceph-deploy purge <gateway-node1> [<gateway-node2>]
ceph-deploy purgedata <gateway-node1> [<gateway-node2>]

If you execute purge, you must re-install Ceph.

As mentioned in the previous section, ceph-deploy places the generated gateway keyring in /var/lib/ceph/radosgw/{rgw-instance} directory for each Ceph Object Gateway node. radosgw-admin utility expects the keyring to be in Ceph configuration directory i.e, /etc/ceph of the Ceph Object Gateway node and will throw error when executed if the keyring isn’t there. So, to ensure radosgw-admin utility runs properly, ssh to each Ceph Object Gateway node and copy the gateway keyring to /etc/ceph/ directory.

For example:

ssh us-east-1
sudo cp /var/lib/ceph/radosgw/ceph-rgw.us-east-1/keyring /etc/ceph
exit

ssh us-west-1
sudo cp /var/lib/ceph/radosgw/ceph-rgw.us-west-1/keyring /etc/ceph
exit

Modify the Ceph configuration file in the /etc/ceph/ directory on the administration node to add an entry for each Ceph Object Gateway node. Add sections entitled [client.rgw.<gateway-node>], replacing <gateway-node> with the short host names of the Ceph Object Gateway nodes of the master region that is, us-east-1 and us-west-1.

Also, Civetweb runs on port 7480 by default. You can change the default port, for example to port 80 by using the rgw_frontends option.

The modified configuration Ceph file will include the following options:

[global]
rgw region root pool = .us.rgw.root

[client.rgw.us-east-1]
rgw region = us
rgw zone = us-east
rgw zone root pool = .us-east.rgw.root
rgw dns name = <hostname>
rgw_frontends = "civetweb port=80"
host = <hostname>

[client.rgw.us-west-1]
rgw region = us
rgw zone = us-west
rgw zone root pool = .us-west.rgw.root
rgw dns name = <hostname>
rgw_frontends = "civetweb port=80"
host = <hostname>

Where, <hostname> is the host name of the Ceph Object Gateway nodes of the master region that is, us-east-1 and us-west-1. To get the host name, run the `hostname -s' command.

The [client.rgw.us-east-1] and [client.rgw.us-west-1] headings identify these portions of the Ceph configuration file as configuring Ceph Storage Cluster clients where each client is a Ceph Object Gateway (that is rgw), and the names of the instances are us-east-1 and us-west-1.

Pull the updated configuration file from /etc/ceph directory to the local directory of your cluster (e.g. ceph-config directory) and push it to your Ceph Object Gateway nodes and other Ceph nodes.

ceph-deploy --overwrite-conf config pull <admin-node>
ceph-deploy --overwrite-conf config push [<gateway-node> ...] [<other-nodes>]

To make the new setting take effect, restart the Ceph Object Gateway on each Ceph Object Gateway node:

sudo systemctl restart ceph-radosgw
sudo chkconfig ceph-radosgw on

Finally, check to ensure that the port you selected is open on the nodes' firewall (e.g., port 80). If it is not open, add the port and reload the firewall configuration on each Ceph Object Gateway node. For example:

sudo firewall-cmd --list-all
sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
sudo firewall-cmd --reload

If you’re running the Ceph Object Gateway on Apache and FastCGI with Red Hat Ceph Storage v1.2.x or above, you’re already running Civetweb—​it starts with the ceph-radosgw daemon and it’s running on port 7480 by default so that it doesn’t conflict with your Apache and FastCGI installation and other commonly used web service ports. Migrating to use Civetweb basically involves removing your Apache installation. Then, you must remove Apache and FastCGI settings from your Ceph configuration file and reset rgw_frontends to Civetweb.

Referring back to the description for installing a Ceph Object Gateway with ceph-deploy, notice that the configuration file has one setting rgw_frontends (and that’s assuming you elected to change the default port). The ceph-deploy utility generates the data directory and the keyring for you—​placing the keyring in /var/lib/ceph/radosgw/{rgw-intance}. The daemon looks in default locations, whereas you may have specified different settings in your Ceph configuration file. Since you already have keys and a data directory, you will want to maintain those paths in your Ceph configuration file if you used something other than default paths.

A typical Ceph Object Gateway configuration file for an Apache-based deployment looks something like this:

[client.radosgw.gateway]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = ""
log file = /var/log/radosgw/client.radosgw.gateway.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
rgw print continue = false

To modify it for use with Civetweb, simply remove the Apache-specific settings such as rgw_socket_path and rgw_print_continue. Then, change the rgw_frontends setting to reflect Civetweb rather than the Apache FastCGI front end and specify the port number you intend to use. For example:

[client.radosgw.gateway]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/radosgw/client.radosgw.gateway.log
rgw_frontends = civetweb port=80

Finally, restart the Ceph Object Gateway.

sudo systemctl restart ceph-radosgw.service

If you used a port number that is not open, you will also have to open that port on your firewall.

Once you finish the setup procedure, if you encounter issues with your configuration, you can add debugging to the [global] section of your Ceph configuration file in /etc/ceph directory of admin node, pull it to the local cluster directory, push it to the gateway nodes and restart the gateway(s) to help troubleshoot any configuration issues. For example:

[global]
#append the following in the global section.
debug ms = 1
debug rgw = 20

To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you need to add a wildcard to the DNS record of the DNS server you use with the ceph-radosgw daemon.

The address of the DNS must also be specified in the Ceph configuration file with the rgw dns name = {hostname} setting. This setting has been mentioned in Add Gateway Configuration to Ceph

This step is important and you will find it’s use in Multi Site Data Replication and Inter-Region Metadata Replication.

For dnsmasq, add the following address setting with a dot (.) prepended to the host name:

address=/.{hostname-or-fqdn}/{host-ip-address}

For example:

address=/.us-west-1/192.168.122.75

For bind, add a wildcard to the DNS record. For example:

$TTL    604800
@       IN      SOA     us-west-1. root.us-west-1. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      us-west-1.
@       IN      A       192.168.122.113
*       IN      CNAME   @

Restart your DNS server and ping your server with a subdomain to ensure that your ceph-radosgw daemon can process the subdomain requests:

ping mybucket.{hostname}

For example:

ping mybucket.us-west-1

A region is designated using a .json file which is used to configure a master region or a secondary region in a Ceph Object Gateway node with specifications for master zone and secondary zone in the region. The region infile is created on a single Ceph Object Gateway node and that Ceph Object Gateway node is also configured as a master zone within that region using an endpoint address. Another Ceph Object Gateway node is added as a secondary zone in the infile using it’s endpoint address. In this section we will create a region infile for master region. Same steps will be repeated for creating a region infile for secondary region in Configuring a Secondary Region, only the name for the .json region file will be different.

To maintain consistency, we will create the region infile in admin node of the cluster and copy it to the desired Ceph Object Gateway nodes.

Execute the following steps for creating and setting up a region:

If you use different Ceph Storage Cluster instances for regions, you should repeat steps 4, 6 and 7 in by executing them with --name client.rgw-us-west-1. You may also export the region map from the initial gateway instance and import it followed by updating the region map.

Like a region, a zone is also divided into two categories, master and secondary. A master zone is basically where you write data, and a secondary zone is where you failover if things go wrong. In this section we will create a zone infile for master zone in us-east-1 Ceph Object Gateway node and a zone infile for secondary zone in us-west-1 Ceph Object Gateway node.

Again, to maintain consistency create the files in admin node of the cluster and copy them to the desired Ceph Object Gateway nodes for master and secondary zones.

Execute the following steps to create and setup zones:

Ceph Object Gateway stores zone users in the zone pools. So you must create zone users after configuring the zones. Copy the access_key and secret_key fields for each zone user so you can update your zone configuration once you complete this step:

ssh us-east-1
sudo radosgw-admin user create --uid="us-east" --display-name="Region-US Zone-East" --name client.rgw.us-east-1 --system
exit

ssh us-west-1
sudo radosgw-admin user create --uid="us-west" --display-name="Region-US Zone-West" --name client.rgw.us-west-1 --system
exit

You must update the zone configuration with zone users so that the synchronization agents can authenticate with the zones.

A Ceph Object Gateway stores bucket index data in the index_pool, which defaults to .rgw.buckets.index. Sometimes users like to put many objects (hundreds of thousands to millions of objects) in a single bucket. If you do not use the gateway administration interface to set quotas for the maximum number of objects per bucket, the bucket index can suffer significant performance degradation when users place large numbers of objects into a bucket.

In Red Hat Ceph Storage v1.3, you may shard bucket indices to help prevent performance bottlenecks when you allow a high number of objects per bucket. The rgw_override_bucket_index_max_shards setting allows you to set a maximum number of shards per bucket. The default value is 0, which means bucket index sharding is off by default.

To turn bucket index sharding on, set rgw_override_bucket_index_max_shards to a value greater than 0.

For federated configurations, each zone may have a different index_pool setting for failover. To make the value consistent for a region’s zones, you may set rgw_override_bucket_index_max_shards in a gateway’s region configuration. For example:

ssh us-east-1
sudo radosgw-admin region get > /etc/ceph/us.json

Here, us.json is the region infile for master region.

Open the us.json file and edit the bucket_index_max_shards setting for each named zone. Save the us.json file and reset the region. For example:

sudo radosgw-admin region set < /etc/ceph/us.json

Once you’ve updated your region, update the region map. For example:

sudo radosgw-admin regionmap update --name client.rgw.us-east-1
exit

Where client.rgw.us-east-1 is the name of the gateway user.

Once you have redeployed your Ceph configuration files, we recommend restarting your Ceph Storage Cluster(s).

Execute from admin node of the cluster:

sudo /etc/init.d/ceph restart

Restart the Ceph Object Gateway daemon on each Ceph Object Gateway node:

# systemctl ceph-radosgw restart

Before configuring the cluster, defining region, zone and instance names will help you manage your cluster. Let’s assume the region represents the European Union, and we refer to it by its standard abbreviation.

Let’s assume the zones represent the Eastern and Western European Union. For continuity, our naming convention will use {region name}-{zone name} format, but you can use any naming convention you prefer.

Finally, let’s assume that zones may have more than one Ceph Object Gateway instance per zone. For continuity, our naming convention will use {region name}-{zone name}-{instance} format, but you can use any naming convention you prefer.

Repeat the exemplary procedure of Configure a Master Region with the following differences:

You need to write and run a Python test script for verifying S3 access. The S3 access test script will connect to the radosgw, create a new bucket and list all buckets. The values for aws_access_key_id and aws_secret_access_key in the Python script are taken from the values of access_key and secret_key returned by the radosgw_admin command while creating the S3 zone users in Create Zone Users.

Execute the following steps on a master or secondary zone Ceph Object Gateway node:

Swift access can be verified via the swift command line client. The command man swift will provide more information on available command line options.

To install swift client, execute the following:

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

To test swift access, you will need the swift_secret_key of a swift zone user.

To create a swift zone user for a particular zone like us-east execute the following:

ssh us-east-1
sudo radosgw-admin subuser create --uid=us-east --subuser=us-east:swift --name client.rgw.us-east-1 --system --access=full

It will create a swift zone user us-east:swift with an access_key. You will need the secret_key for this user as well.

To generate a secret key for the swift user, execute:

sudo radosgw-admin key create --subuser=us-east:swift --name client.rgw.us-east-1 --system --key-type=swift --gen-secret

To test swift access, execute the following:

swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list

Replace {IP ADDRESS} with the public IP address of the gateway server and {swift_secret_key} with its value from the output of radosgw-admin key create command executed for the swift user. Replace {port} with the port number you are using with Civetweb (e.g., 7480 is the default). If you are using port 80 for Civetweb, there is no need to provide {port}.

For example:

swift -A http://10.19.143.116:7480/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list

The output should be:

my-new-bucket