Chapter 2. Installation
For Red Hat Ceph Storage v1.3, Red Hat supports the Ceph Object Gateway running on Civetweb (embedded into the radosgw
daemon) instead of Apache and FastCGI. Using Civetweb simplifies the installation and configuration.
To run the Ceph Object Gateway service On Ubuntu Trusty
14.04, you should have a running Ceph storage cluster, and the Ceph Object Gateway node should have access to the public network.
In version 1.3, the Ceph Object Gateway does not support SSL. You may setup a reverse proxy server with SSL to dispatch HTTPS requests as HTTP requests to CivetWeb.
2.1. Prerequisites
Refer to the Red Hat Ceph Storage 1.3 Installation Guide for Ubuntu (x86_64)
and execute the pre-installation procedures on your Ceph Object Gateway node. Specifically, you should set up a Ceph Deploy user with password-less sudo
. For Ceph Object Gateways, you will need to open the port that Civetweb will use in production.
Civetweb runs on port 7480
by default.
2.2. Installation Types
Starting from version 1.3.1, Red Hat Ceph Storage supports the use of online repositories from https://rhcs.download.redhat.com/ubuntu. So, for RHCS v1.3.2, you can install repository in the Ceph Object Gateway node in two ways:
- ISO based installation
- Online repository based installation
2.2.1. ISO Based Installation
In ISO based installation, to install and setup Ceph Object Gateway on the Ceph Object Gateway node, you will need either ceph-mon
or ceph-osd
repository. Both of these repositories contain the radosgw
package. So, you can use anyone of them for the Ceph Object Gateway node.
To install ceph-mon
or ceph-osd
repository on Ceph Object Gateway node, execute the following from admin
node:
ceph-deploy repo [ceph-mon|ceph-osd] <gateway-node>
For example:
ceph-deploy repo ceph-mon <gateway-node>
The above command will write a .list
file on the Ceph Object Gateway node pointing to the local repositories created on the admin node by ice_setup
. This will enable the Ceph Object Gateway node to receive packages from local repositories.
2.2.2. Online Repository Based Installation
In online repository based installation, to install and setup Ceph Object Gateway on the Ceph Object Gateway node, you will need Tools
repository.
To install Tools
repository in Ceph Object Gateway node, execute the following from admin
node:
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Tools' --gpg-url https://www.redhat.com/security/fd431d51.txt Tools <gateway-node>
To execute the above command, your admin
node should be properly setup with the required online repositories and the latest version of ceph-deploy
for RHCS v1.3.1 should be installed. See the Set Online Ceph Repositories
section in Red Hat Ceph Storage 1.3 Installation Guide for Ubuntu (x86_64)
for details.
2.3. Installing the Object Gateway
From the working directory of your administration server, install the Ceph Object Gateway package on the Ceph Object Gateway node. For example:
ceph-deploy install --rgw <gateway-node1> [<gateway-node2> ...]
The ceph-common
package is a dependency, so ceph-deploy
will install this too. The ceph
CLI tools are intended for administrators. To make your Ceph Object Gateway node an administrator node, execute the following from the working directory of your administration server.
ceph-deploy admin <node-name>
2.4. Create a Gateway Instance
From the working directory of your administration server, execute the following to create an instance of the Ceph Object Gateway on the Ceph Object Gateway:
ceph-deploy rgw create <gateway-node1>
Once the gateway is running, you should be able to access it on port 7480
with an unauthenticated request in a web browser or a command line HTTP client (e.g., curl
, wget
etc.) like this:
http://gateway-node1:7480
If the gateway instance is working properly, your browser or CLI HTTP client should receive a response like this:
<?xml version="1.0" encoding="UTF-8"?> <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID>anonymous</ID> <DisplayName></DisplayName> </Owner> <Buckets> </Buckets> </ListAllMyBucketsResult>
If at any point you run into trouble and you want to start over, execute the following from the working directory of your administration server to purge the configuration:
ceph-deploy purge <gateway-node1> [<gateway-node2>] ceph-deploy purgedata <gateway-node1> [<gateway-node2>]
If you execute purge
, you must re-install Ceph.
2.5. Changing Your Default Port
Civetweb runs on port 7480
by default. To change the default port (e.g,. to port 80
), modify your Ceph configuration file in /etc/ceph
directory of your administration server. Add a section entitled [client.rgw.<gateway-node>]
, replacing <gateway-node>
with the short host name of your Ceph Object Gateway node (i.e., hostname -s
).
In version 1.3, the Ceph Object Gateway does not support SSL. You may setup a reverse proxy web server with SSL to dispatch HTTPS requests as HTTP requests to CivetWeb.
For example, if your node name is gateway-node
, add a section like this after the [global]
section:
[client.rgw.gateway-node] rgw_frontends = "civetweb port=80"
Ensure that you leave no whitespace between port=<port-number>
in the rgw_frontends
key/value pair. The [client.rgw.gateway-node]
heading identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (i.e., rgw
), and the name of the instance is gateway-node
.
Pull the updated configuration file from /etc/ceph
directory to the working directory of your administration server (e.g. ceph-config directory) and push it to your Ceph Object Gateway node and other Ceph nodes.
ceph-deploy --overwrite-conf config pull <admin-node> ceph-deploy --overwrite-conf config push <gateway-node> [<other-nodes>]
Stop Apache so that Ceph Object Gateway can run on port 80
.
sudo service apache2 stop
To make the new port setting take effect, restart the Ceph Object Gateway.
sudo service radosgw restart id=rgw.<short-hostname>
Finally, check to ensure that the port you selected is open on the node’s firewall (e.g., port 80
). If it is not open, add the port and reload the firewall configuration. For example:
sudo iptables --list sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
Replace <iface>
and <ip-address>/<netmask>
with the relevant values for your Ceph Object Gateway node.
Once you have finished configuring iptables
, ensure that you make the change persistent so that it will be in effect when your Ceph Object Gateway node reboots.
Execute:
sudo apt-get install iptables-persistent
A terminal UI will open up. Select yes
for the prompts to save current IPv4
iptables rules to /etc/iptables/rules.v4
and current IPv6
iptables rules to /etc/iptables/rules.v6
.
The IPv4
iptables rule that you set in the earlier step will be loaded in /etc/iptables/rules.v4
and will be persistent across reboots.
If you add a new IPv4
iptables rule after installing iptables-persistent
you will have to add it to the rule file. In such case, execute the following as a root
user:
iptables-save > /etc/iptables/rules.v4
2.6. Migrating from Apache to Civetweb
If you’re running the Ceph Object Gateway on Apache and FastCGI with Red Hat Ceph Storage v1.2.x or above, you’re already running Civetweb. It starts with the radosgw
daemon and it’s running on port 7480 by default so that it doesn’t conflict with your Apache and FastCGI installation and other commonly used web service ports. Migrating to use Civetweb basically involves removing your Apache installation. Then, you must remove Apache and FastCGI settings from your Ceph configuration file and reset rgw_frontends
to Civetweb.
Referring back to the description for installing a Ceph Object Gateway with ceph-deploy
, notice that the configuration file only has one setting rgw_frontends
(and that’s assuming you elected to change the default port). The ceph-deploy
utility generates the data directory and the keyring for you, placing the keyring in /var/lib/ceph/radosgw/{rgw-intance}
. The daemon looks in default locations, whereas you may have specified different settings in your Ceph configuration file. Since you already have keys and a data directory, you will want to maintain those paths in your Ceph configuration file if you used something other than default paths.
A typical Ceph Object Gateway configuration file for an Apache-based deployment looks something like this:
[client.radosgw.gateway-node] host = {hostname} keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /var/log/radosgw/client.radosgw.gateway-node1.log
To modify it for use with Civetweb, simply remove the Apache-specific setting rgw_socket_path
. Then, change the rgw_frontends
setting to reflect Civetweb rather than the Apache FastCGI front end and specify the port number you intend to use. For example:
[client.radosgw.gateway-node] host = {hostname} keyring = /etc/ceph/ceph.client.radosgw.keyring log file = /var/log/radosgw/client.radosgw.gateway-node1.log rgw_frontends = civetweb port=80
Finally, restart the Ceph Object Gateway.
sudo service radosgw restart id=rgw.<short-hostname>
If you used a port number that is not open, you will also need to open that port on your firewall.
2.7. Adding a Wildcard to DNS
To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you need to add a wildcard to the DNS record of the DNS server you use with the ceph-radosgw
daemon.
The address of the DNS must also be specified in the Ceph configuration file with the rgw dns name = {hostname}
setting.
For dnsmasq
, add the following address setting with a dot (.) prepended to the host name:
address=/.{hostname-or-fqdn}/{host-ip-address}
For example:
address=/.gateway-node1/192.168.122.75
For bind
, add a wildcard to the DNS record. For example:
$TTL 604800 @ IN SOA gateway-node1. root.gateway-node1. ( 2 ; Serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 604800 ) ; Negative Cache TTL ; @ IN NS gateway-node1. @ IN A 192.168.122.113 * IN CNAME @
Restart your DNS server and ping your server with a subdomain to ensure that your ceph-radosgw
daemon can process the subdomain requests:
ping mybucket.{hostname}
For example:
ping mybucket.gateway-node1
2.8. Adjusting Logging and Debugging Output
Once you finish the setup procedure, check your logging output to ensure it meets your needs. Log files are located in /var/log/radosgw
by default. If you encounter issues with your configuration, you can increase logging and debugging messages in the [global]
section of your Ceph configuration file and restart the gateway(s) to help troubleshoot any configuration issues. For example:
[global] #append the following in the global section. debug ms = 1 debug rgw = 20 debug civetweb = 20
You may also modify these settings at runtime. For example:
ceph tell osd.0 injectargs --debug_civetweb 10/20
For general details on logging and debugging, see Logging and Debugging. For Ceph Object Gateway-specific details on logging settings, see Logging Settings in this guide.
2.9. Using the Object Gateway
To use the REST interfaces, first create an initial Ceph Object Gateway user for the S3 interface. Then, create a subuser for the Swift interface. You then need to verify if the created users are able to access the gateway.
2.9.1. Create a radosgw
User for S3 Access
A radosgw
user needs to be created and granted access. The command man radosgw-admin
will provide information on additional command options.
To create the user, execute the following on the gateway host
:
sudo radosgw-admin user create --uid="testuser" --display-name="First User"
The output of the command will be something like the following:
{ "user_id": "testuser", "display_name": "First User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [{ "user": "testuser", "access_key": "I0PJDPCIYZ665MW88W9R", "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" }], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] }
The values of keys→access_key
and keys→secret_key
are needed for access validation.
Check the key output. Sometimes radosgw-admin
generates a JSON escape character \
in access_key
or secret_key
and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character \
, encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually. Also, if radosgw-admin
generates a JSON escape character \
and a forward slash /
together in a key, like \/
, only remove the JSON escape character \
. Do not remove the forward slash /
as it is a valid character in the key.
2.9.2. Create a Swift User
A Swift subuser needs to be created if this kind of access is needed. Creating a Swift user is a two step process. The first step is to create the user. The second is to create the secret key.
Execute the following steps on the gateway host
:
Create the Swift user:
sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
The output will be something like the following:
{ "user_id": "testuser", "display_name": "First User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [{ "id": "testuser:swift", "permissions": "full-control" }], "keys": [{ "user": "testuser:swift", "access_key": "3Y1LNW4Q6X0Y53A52DET", "secret_key": "" }, { "user": "testuser", "access_key": "I0PJDPCIYZ665MW88W9R", "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" }], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] }
Create the secret key:
sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
The output will be something like the following:
{ "user_id": "testuser", "display_name": "First User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [{ "id": "testuser:swift", "permissions": "full-control" }], "keys": [{ "user": "testuser:swift", "access_key": "3Y1LNW4Q6X0Y53A52DET", "secret_key": "" }, { "user": "testuser", "access_key": "I0PJDPCIYZ665MW88W9R", "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" }], "swift_keys": [{ "user": "testuser:swift", "secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA" }], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] }
2.9.3. Test S3 Access
You need to write and run a Python test script for verifying S3 access. The S3 access test script will connect to the radosgw
, create a new bucket and list all buckets. The values for aws_access_key_id
and aws_secret_access_key
are taken from the values of access_key
and secret_key
returned by the radosgw_admin
command.
Execute the following steps:
You will need to install the
python-boto
package.sudo apt-get install python-boto
Create the Python script:
vi s3test.py
Add the following contents to the file:
import boto import boto.s3.connection access_key = 'I0PJDPCIYZ665MW88W9R' secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA' conn = boto.connect_s3( aws_access_key_id = access_key, aws_secret_access_key = secret_key, host = '{FQDN}', port = {port}, is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), ) bucket = conn.create_bucket('my-new-bucket') for bucket in conn.get_all_buckets(): print "{name}\t{created}".format( name = bucket.name, created = bucket.creation_date, )
Replace
{FQDN}
with the full hostname i.e, the fully qualified domain name of Ceph Object Gateway node. Replace {port} with the port number you are using with Civetweb.Run the script:
python s3test.py
The output will be something like the following:
my-new-bucket 2015-02-16T17:09:10.000Z
2.9.4. Test Swift Access
Swift access can be verified via the swift
command line client. The command man swift
will provide more information on available command line options.
To install swift
client, execute the following:
sudo apt-get install python-setuptools sudo easy_install pip sudo pip install --upgrade setuptools sudo pip install --upgrade python-swiftclient
To test swift access, execute the following:
swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list
Replace {IP ADDRESS}
with the public IP address of the gateway server and {swift_secret_key}
with its value from the output of radosgw-admin key create
command executed for the swift
user. Replace {port} with the port number you are using with Civetweb (e.g., 7480
is the default). If you don’t replace the port, it will default to port 80
.
For example:
swift -A http://10.19.143.116:7480/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list
The output should be:
my-new-bucket