Chapter 2. Object Gateway S3 Application Programming Interface (API)
Red Hat Ceph Object Gateway supports a RESTful API that is compatible with the basic data access model of the Amazon S3 API.
The following table describes the support status for current Amazon S3 functional features.
Feature | Status | Remarks |
---|---|---|
Supported | ||
Supported | Different set of canned ACLs | |
Partially Supported |
| |
Supported | ||
Partially Supported |
| |
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | Different set of canned ACLs | |
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Partially Supported |
| |
Partially Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported | ||
Supported |
The following table lists the common request header fields that are not supported.
Name | Type |
---|---|
x-amz-security-token | Request |
Server | Response |
x-amz-delete-marker | Response |
x-amz-id-2 | Response |
x-amz-request-id | Response |
x-amz-version-id | Response |
2.1. S3 API Server-side Encryption
The Ceph Object Gateway supports server-side encryption of uploaded objects for the S3 API. Server-side encryption means that the S3 client sends data over HTTP in its unencrypted form, and the Ceph Object Gateway stores that data in the Ceph Storage Cluster in encrypted form.
Red Hat does NOT support S3 object encryption of SLO(Static Large Object) and DLO(Dynamic Large Object).
To use encryption, client requests MUST send requests over an SSL connection. Red Hat does not support S3 encryption from a client unless the Ceph Object Gateway uses SSL. However, for testing purposes, administrators may disable SSL during testing by setting the rgw_crypt_require_ssl
configuration setting to false
at runtime, setting it to false
in the Ceph configuration file and restarting the gateway instance, or setting it to false
in the Ansible configuration files and replaying the Ansible playbooks for the Ceph Object Gateway.
There are two options for the management of encryption keys:
Customer-Provided Keys
When using customer-provided keys, the S3 client passes an encryption key along with each request to read or write encrypted data. It is the customer’s responsibility to manage those keys. Customers must remember which key the Ceph Object Gateway used to encrypt each object.
Ceph Object Gateway implements the customer-provided key behavior in the S3 API according to the Amazon SSE-C specification.
Since the customer handles the key management and the S3 client passes keys to the Ceph Object Gateway, the Ceph Object Gateway requires no special configuration to support this encryption mode.
Key Management Service
When using a key management service, the secure key management service stores the keys and the Ceph Object Gateway retrieves them on demand to serve requests to encrypt or decrypt data.
Ceph Object Gateway implements the key management service behavior in the S3 API according to the Amazon SSE-KMS specification.
Currently, the only tested key management implementation uses OpenStack Barbican. However, OpenStack Barbican is a Technology Preview and is not supported for use in production systems.
2.2. Bucket Policies
The Ceph Object Gateway supports a subset of the Amazon S3 policy language applied to buckets.
Creation and Removal
Ceph Object Gateway manages S3 Bucket policies through standard S3 operations rather than using the radosgw-admin
CLI tool.
Administrators may use the s3cmd
command to set or delete a policy. For example:
$ cat > examplepol { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred"]}, "Action": "s3:PutObjectAcl", "Resource": [ "arn:aws:s3:::happybucket/*" ] }] } $ s3cmd setpolicy examplepol s3://happybucket $ s3cmd delpolicy s3://happybucket
Limitations
Ceph Object Gateway only supports the following S3 actions:
-
s3:AbortMultipartUpload
-
s3:CreateBucket
-
s3:DeleteBucketPolicy
-
s3:DeleteBucket
-
s3:DeleteBucketWebsite
-
s3:DeleteObject
-
s3:DeleteObjectVersion
-
s3:GetBucketAcl
-
s3:GetBucketCORS
-
s3:GetBucketLocation
-
s3:GetBucketPolicy
-
s3:GetBucketRequestPayment
-
s3:GetBucketVersioning
-
s3:GetBucketWebsite
-
s3:GetLifecycleConfiguration
-
s3:GetObjectAcl
-
s3:GetObject
-
s3:GetObjectTorrent
-
s3:GetObjectVersionAcl
-
s3:GetObjectVersion
-
s3:GetObjectVersionTorrent
-
s3:ListAllMyBuckets
-
s3:ListBucketMultiPartUploads
-
s3:ListBucket
-
s3:ListBucketVersions
-
s3:ListMultipartUploadParts
-
s3:PutBucketAcl
-
s3:PutBucketCORS
-
s3:PutBucketPolicy
-
s3:PutBucketRequestPayment
-
s3:PutBucketVersioning
-
s3:PutBucketWebsite
-
s3:PutLifecycleConfiguration
-
s3:PutObjectAcl
-
s3:PutObject
-
s3:PutObjectVersionAcl
Ceph Object Gateway does not support setting policies on users, groups, or roles.
The Ceph Object Gateway uses the RGW ‘tenant’ identifier in place of the Amazon twelve-digit account ID. Ceph Object Gateway administrators who want to use policies between Amazon Web Service (AWS) S3 and Ceph Object Gateway S3 will have to use the Amazon account ID as the tenant ID when creating users.
With AWS S3, all tenants share a single namespace. By contrast, Ceph Object Gateway gives every tenant its own namespace of buckets. At present, Ceph Object Gateway clients trying to access a bucket belonging to another tenant MUST address it as tenant:bucket
in the S3 request.
In the AWS, a bucket policy can grant access to another account, and that account owner can then grant access to individual users with user permissions. Since Ceph Object Gateway does not yet support user, role, and group permissions, account owners will need to grant access directly to individual users.
Granting an entire account access to a bucket grants access to ALL users in that account.
Bucket policies do NOT support string interpolation.
Ceph Object Gateway supports the following condition keys:
-
aws:CurrentTime
-
aws:EpochTime
-
aws:PrincipalType
-
aws:Referer
-
aws:SecureTransport
-
aws:SourceIp
-
aws:UserAgent
-
aws:username
Ceph Object Gateway ONLY supports the following condition keys for the ListBucket
action:
-
s3:prefix
-
s3:delimiter
-
s3:max-keys
Impact on Swift
Ceph Object Gateway provides no functionality to set bucket policies under the Swift API. However, bucket policies that have been set with the S3 API govern Swift as well as S3 operations.
Ceph Object Gateway matches Swift credentials against Principals specified in a policy.
2.3. Authentication and Access Control Lists
Requests to the Ceph Object Gateway can be either authenticated or unauthenticated. Ceph Object Gateway assumes unauthenticated requests are sent by an anonymous user. Ceph Object Gateway supports canned ACLs.
2.3.1. Authentication
For most use cases, clients use existing open source libraries like the Amazon SDK’s AmazonS3Client
for Java, and Python Boto, where you simply pass in the access key and secret key, and the library builds the request header and authentication signature for you. However, you can create requests and sign them too.
Authenticating a request requires including an access key and a base 64-encoded Hash-based Message Authentication Code (HMAC) in the request before it is sent to the Ceph Object Gateway server. Ceph Object Gateway uses an S3-compatible authentication approach.
Example
HTTP/1.1 PUT /buckets/bucket/object.mpeg Host: cname.domain.com Date: Mon, 2 Jan 2012 00:01:01 +0000 Content-Encoding: mpeg Content-Length: 9999999 Authorization: AWS <access_key>:<hash_of_header_and_secret>
In the foregoing example, replace <access_key>
with the value for the access key ID followed by a colon (:
). Replace <hash_of_header_and_secret>
with a hash of a canonicalized header string and the secret corresponding to the access key ID.
To generate the hash of the header string and secret, you must:
- Get the value of the header string.
- Normalize the request header string into canonical form.
- Generate an HMAC using a SHA-1 hashing algorithm.
-
Encode the
hmac
result as base-64.
To normalize the header into canonical form:
-
Get all
content-
headers. -
Remove all
content-
headers except forcontent-type
andcontent-md5
. -
Ensure the
content-
header names are lowercase. -
Sort the
content-
headers lexicographically. -
Ensure you have a
Date
header AND ensure the specified date uses GMT and not an offset. -
Get all headers beginning with
x-amz-
. -
Ensure that the
x-amz-
headers are all lowercase. -
Sort the
x-amz-
headers lexicographically. - Combine multiple instances of the same field name into a single field and separate the field values with a comma.
- Replace white space and line breaks in header values with a single space.
- Remove white space before and after colons.
- Append a new line after each header.
- Merge the headers back into the request header.
Replace the <hash_of_header_and_secret>
with the base-64 encoded HMAC string.
For additional details, consult the Signing and Authenticating REST Requests section of Amazon Simple Storage Service documentation.
2.3.2. Access Control Lists (ACLs)
Ceph Object Gateway supports S3-compatible ACL functionality. An ACL is a list of access grants that specify which operations a user can perform on a bucket or on an object. Each grant has a different meaning when applied to a bucket versus applied to an object:
Permission | Bucket | Object |
---|---|---|
| Grantee can list the objects in the bucket. | Grantee can read the object. |
| Grantee can write or delete objects in the bucket. | N/A |
| Grantee can read bucket ACL. | Grantee can read the object ACL. |
| Grantee can write bucket ACL. | Grantee can write to the object ACL. |
| Grantee has full permissions for object in the bucket. | Grantee can read or write to the object ACL. |
2.3.3. Authentication using the STS Lite API (Technology Preview)
The Ceph Object Gateway provides support for a subset of the Amazon Secure Token Service (STS) REST APIs. STS Lite provides access to a set of temporary credentials for identity and access management.
The STS Lite authentication mechanism is integrated with Keystone in the Ceph Object Gateway. A set of temporary security credentials are returned after authenticating a set of Amazon Web Services (AWS) credentials with Keystone. The STS engine authenticates these temporary security credentials made by subsequent S3 calls, resulting in less load on the Keystone server.
The Ceph Object Gateway implements the following STS Lite REST APIs:
- GetSessionToken
Returns a set of temporary security credentials for a set of AWS credentials. Use this API for initial authentication with Keystone and the temporary credentials returned can be used to make subsequent S3 calls. The temporary credentials will have the same permission as that of the AWS credentials.
Parameters:
- DurationSeconds (Integer/ Optional)
-
The duration in seconds for which the credentials should remain valid. The default value is 3600 seconds. The max value is 43200 seconds. You can set this value using the
rgw_sts_max_session
option in the Ceph configuration file, by default the/etc/ceph/ceph.conf
file. - SerialNumber (String/ Optional)
- The identifier number of the Multi-Factor Authentication (MFA) device associated with the user making the GetSessionToken call.
- TokenCode (String/ Optional)
- The value provided by the MFA device, if MFA is required.
- AssumeRole
Returns a set of temporary credentials that can be used for cross-account access. The temporary credentials will have permissions that are allowed by both - permission policies attached with the Role and policy attached with the AssumeRole API.
Parameters:
- RoleArn (String/ Required)
- Amazon Resource Name (ARN) of the Role to Assume.
- RoleSessionName (String/ Required)
An Identifier for the assumed role session.
- Policy (String/ Optional):
- An IAM Policy in JSON format.
- DurationSeconds (Integer/ Optional)
- The duration in seconds of the session. The default value is 3600.
- ExternalId (String/ Optional)
- A unique Id that might be used when a role is assumed in another account.
- SerialNumber (String/ Optional)
- The identifier number of the MFA device associated with the user making the AssumeRole call.
- TokenCode (String/ Optional)
- The value provided by the MFA device, if the trust policy of the role being assumed requires MFA.
Additional Resources
- See the AWS Security Token Service API Reference documentation for more information.
- See the AWS Identity and Access Management User Guide for more information.
2.4. Accessing the Gateway
You can use various programming languages to create a connection with the gateway server and do the bucket management tasks. There are different open source libraries available for these programming languages that are used for authentication with the gateway.
The sections mentioned below will describe the procedure for some of the popular programming languages.
2.4.1. Prerequisites
You must follow some prerequisites on the Ceph Object Gateway node before attempting to access the gateway server. The prerequisites are as follows:
Set up the gateway server properly by following the instructions based on the operating system:
- For Red Hat Enterprise Linux, see the Ceph Object Gateway Installation chapter in the Installation Guide for Red hat Enterprise Linux.
- For Ubuntu, see the Ceph Object Gateway Installation chapter in the Installation Guide for Ubuntu.
-
DO NOT modify the Ceph configuration file to use port
80
and letCivetweb
use the default Ansible configured port of8080
. As
root
, open port8080
on firewall:# firewall-cmd --zone=public --add-port=8080/tcp --permanent # firewall-cmd --reload
Add a wildcard to the DNS server that you are using for the gateway as mentioned in the Object Gateway Guide.
You can also set up the gateway node for local DNS caching. To do so, execute the following steps:
As
root
, install and setupdnsmasq
:# yum install dnsmasq # echo "address=/.<FQDN_of_gateway_node>/<IP_of_gateway_node>" | tee --append /etc/dnsmasq.conf # systemctl start dnsmasq # systemctl enable dnsmasq
Replace
<IP_of_gateway_node>
and<FQDN_of_gateway_node>
with the IP address and FQDN of the gateway node.As
root
, stop NetworkManager:# systemctl stop NetworkManager # systemctl disable NetworkManager
As
root
, set the gateway server’s IP as the nameserver:# echo "DNS1=<IP_of_gateway_node>" | tee --append /etc/sysconfig/network-scripts/ifcfg-eth0 # echo "<IP_of_gateway_node> <FQDN_of_gateway_node>" | tee --append /etc/hosts # systemctl restart network # systemctl enable network # systemctl restart dnsmasq
Replace
<IP_of_gateway_node>
and<FQDN_of_gateway_node>
with the IP address and FQDN of the gateway node.Verify subdomain requests:
$ ping mybucket.<FQDN_of_gateway_node>
Replace
<FQDN_of_gateway_node>
with the FQDN of the gateway node.WarningSetting up the gateway server for local DNS caching is for testing purposes only. You won’t be able to access outside network after doing this. It is strongly recommended to use a proper DNS server for the Ceph cluster and gateway node.
-
Create the
radosgw
user forS3
access carefully as mentioned in the Object Gateway Guide for Red Hat Enterprise Linux or Object Gateway Guide for Ubuntu and copy the generatedaccess_key
andsecret_key
. You will need these keys forS3
access and subsequent bucket management tasks.
2.4.2. Ruby AWS::S3 Examples (aws-s3 gem)
You can use Ruby programming language along with aws-s3
gem for S3
access. Execute the steps mentioned below on the node used for accessing the Ceph Object Gateway server with Ruby AWS::S3
.
Setup Ruby
Execute the following steps to setup Ruby:
As
root
, installruby
:# yum install ruby
NoteThe above command will install
ruby
and it’s essential dependencies likerubygems
andruby-libs
too. If somehow the command doesn’t install all the dependencies, install them separately.As
root
, installaws-s3
:# gem install aws-s3
Creating a connection
Create a project directory:
$ mkdir ruby_aws_s3 $ cd ruby_aws_s3
Create the connection file:
$ vim conn.rb
Paste the following contents into the
conn.rb
file:#!/usr/bin/env ruby require 'aws/s3' require 'resolv-replace' AWS::S3::Base.establish_connection!( :server => '<FQDN_of_gateway_node>', :port => '8080', :access_key_id => 'my-access-key', :secret_access_key => 'my-secret-key' )
Replace
<FQDN_of_gateway_node>
with the FQDN of your gateway node. Replacemy-access-key
andmy-secret-key
with theaccess_key
andsecret_key
that were generated when you created theradosgw
user forS3
access as mentioned in the Object Gateway Guide for Red Hat Enterprise Linux or the Object Gateway Guide for Ubuntu.An example connection file looks like the following:
#!/usr/bin/env ruby require 'aws/s3' require 'resolv-replace' AWS::S3::Base.establish_connection!( :server => 'testclient.englab.pnq.redhat.com', :port => '8080', :access_key_id => '98J4R9P22P5CDL65HKP8', :secret_access_key => '6C+jcaP0dp0+FZfrRNgyGA9EzRy25pURldwje049' )
Save the file and exit the editor.
Make the file executable:
$ chmod +x conn.rb
Run the file:
$ ./conn.rb | echo $?
If you have provided the values correctly in the file, the output of the command will be
0
.
Creating a bucket
Create a new file:
$ vim create_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::Bucket.create('my-new-bucket1')
Save the file and exit the editor.
Make the file executable:
$ chmod +x create_bucket.rb
Run the file:
$ ./create_bucket.rb
If the output of the command is
true
it would mean that bucketmy-new-bucket1
was created successfully.
Listing owned buckets
Create a new file:
$ vim list_owned_buckets.rb
Paste the following content into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::Service.buckets.each do |bucket| puts "#{bucket.name}\t#{bucket.creation_date}" end
Save the file and exit the editor.
Make the file executable:
$ chmod +x list_owned_buckets.rb
Run the file:
$ ./list_owned_buckets.rb
The output should look something like this:
my-new-bucket1 2016-01-21 10:33:19 UTC
Creating an object
Create a new file:
$ vim create_object.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::S3Object.store( 'hello.txt', 'Hello World!', 'my-new-bucket1', :content_type => 'text/plain' )
Save the file and exit the editor.
Make the file executable:
$ chmod +x create_object.rb
Run the file:
$ ./create_object.rb
This will create a file
hello.txt
with the stringHello World!
.
Listing a Bucket’s Content
Create a new file:
$ vim list_bucket_content.rb
Paste the following content into the file:
#!/usr/bin/env ruby load 'conn.rb' new_bucket = AWS::S3::Bucket.find('my-new-bucket1') new_bucket.each do |object| puts "#{object.key}\t#{object.about['content-length']}\t#{object.about['last-modified']}" end
Save the file and exit the editor.
Make the file executable.
$ chmod +x list_bucket_content.rb
Run the file:
$ ./list_bucket_content.rb
The output will look something like this:
hello.txt 12 Fri, 22 Jan 2016 15:54:52 GMT
Deleting a empty bucket
Create a new file:
$ vim del_empty_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::Bucket.delete('my-new-bucket1')
Save the file and exit the editor.
Make the file executable:
$ chmod +x del_empty_bucket.rb
Run the file:
$ ./del_empty_bucket.rb | echo $?
If the bucket is successfully deleted, the command will return
0
as output.NotePlease edit the
create_bucket.rb
file to create empty buckets likemy-new-bucket9
,my-new-bucket10
etc and edit the above mentioneddel_empty_bucket.rb
file accordingly before trying to delete empty buckets.
Deleting a non-empty bucket (forcefully)
Create a new file:
$ vim del_non_empty_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::Bucket.delete('my-new-bucket1', :force => true)
Save the file and exit the editor.
Make the file executable:
$ chmod +x del_non_empty_bucket.rb
Run the file:
$ ./del_non_empty_bucket.rb | echo $?
If the bucket is successfully deleted, the command will return
0
as output.
Deleting an object
Create a new file:
$ vim delete_object.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' AWS::S3::S3Object.delete('hello.txt', 'my-new-bucket1')
Save the file and exit the editor.
Make the file executable:
$ chmod +x delete_object.rb
Run the file:
$ ./delete_object.rb
This will delete the object
hello.txt
.
2.4.3. Ruby AWS::SDK Examples (aws-sdk gem ~>2)
You can use the Ruby programming language along with aws-sdk
gem for S3
access. Execute the steps mentioned below on the node used for accessing the Ceph Object Gateway server with Ruby AWS::SDK
.
Setup Ruby
Execute the following steps to setup Ruby:
As
root
, installruby
:# yum install ruby
NoteThe above command will install
ruby
and its essential dependencies such asrubygems
andruby-libs
. If somehow the command doesn’t install all the dependencies, install them separately.As
root
, installaws-sdk
:# gem install aws-sdk
Creating a connection
Create a project directory:
$ mkdir ruby_aws_sdk $ cd ruby_aws_sdk
Create the connection file:
$ vim conn.rb
Paste the following contents into the
conn.rb
file:#!/usr/bin/env ruby require 'aws-sdk' require 'resolv-replace' Aws.config.update( endpoint: 'http://<FQDN_of_gateway_node>:8080', access_key_id: 'my-access-key', secret_access_key: 'my-secret-key', force_path_style: true, region: 'us-east-1' )
Replace
<FQDN_of_gateway_node>
with the FQDN of your gateway node. Replacemy-access-key
andmy-secret-key
with theaccess_key
andsecret_key
that were generated when you created theradosgw
user forS3
access as mentioned in the Red Hat Ceph Storage Object Gateway Guide.An example connection file looks like the following:
#!/usr/bin/env ruby require 'aws-sdk' require 'resolv-replace' Aws.config.update( endpoint: 'http://testclient.englab.pnq.redhat.com:8080', access_key_id: '98J4R9P22P5CDL65HKP8', secret_access_key: '6C+jcaP0dp0+FZfrRNgyGA9EzRy25pURldwje049', force_path_style: true, region: 'us-east-1' )
Save the file and exit the editor.
Make the file executable:
chmod +x conn.rb
Run the file:
./conn.rb | echo $?
If you have provided the values correctly in the file, the output of the command will be
0
.
Creating a bucket
Create a new file:
vim create_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.create_bucket(bucket: 'my-new-bucket2')
Save the file and exit the editor.
Make the file executable:
chmod +x create_bucket.rb
Run the file:
./create_bucket.rb
If the output of the command is
true
it would mean that bucketmy-new-bucket2
was created successfully.
Listing owned buckets
Create a new file:
vim list_owned_buckets.rb
Paste the following content into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.list_buckets.buckets.each do |bucket| puts "#{bucket.name}\t#{bucket.creation_date}" end
Save the file and exit the editor.
Make the file executable:
chmod +x list_owned_buckets.rb
Run the file:
./list_owned_buckets.rb
The output should look something like this:
my-new-bucket2 2016-01-21 10:33:19 UTC
Creating an object
Create a new file:
vim create_object.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.put_object( key: 'hello.txt', body: 'Hello World!', bucket: 'my-new-bucket2', content_type: 'text/plain' )
Save the file and exit the editor.
Make the file executable:
chmod +x create_object.rb
Run the file:
./create_object.rb
This will create a file
hello.txt
with the stringHello World!
.
Listing a Bucket’s Content
Create a new file:
vim list_bucket_content.rb
Paste the following content into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.list_objects(bucket: 'my-new-bucket2').contents.each do |object| puts "#{object.key}\t#{object.size}" end
Save the file and exit the editor.
Make the file executable.
chmod +x list_bucket_content.rb
Run the file:
./list_bucket_content.rb
The output will look something like this:
hello.txt 12 Fri, 22 Jan 2016 15:54:52 GMT
Deleting a empty bucket
Create a new file:
vim del_empty_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.delete_bucket(bucket: 'my-new-bucket2')
Save the file and exit the editor.
Make the file executable:
chmod +x del_empty_bucket.rb
Run the file:
./del_empty_bucket.rb | echo $?
If the bucket is successfully deleted, the command will return
0
as output.NotePlease edit the
create_bucket.rb
file to create empty buckets likemy-new-bucket6
,my-new-bucket7
etc and edit the above mentioneddel_empty_bucket.rb
file accordingly before trying to delete empty buckets.
Deleting a non-empty bucket (forcefully)
Create a new file:
vim del_non_empty_bucket.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new Aws::S3::Bucket.new('my-new-bucket2', client: s3_client).clear! s3_client.delete_bucket(bucket: 'my-new-bucket2')
Save the file and exit the editor.
Make the file executable:
chmod +x del_non_empty_bucket.rb
Run the file:
./del_non_empty_bucket.rb | echo $?
If the bucket is successfully deleted, the command will return
0
as output.
Deleting an object
Create a new file:
vim delete_object.rb
Paste the following contents into the file:
#!/usr/bin/env ruby load 'conn.rb' s3_client = Aws::S3::Client.new s3_client.delete_object(key: 'hello.txt', bucket: 'my-new-bucket2')
Save the file and exit the editor.
Make the file executable:
chmod +x delete_object.rb
Run the file:
./delete_object.rb
This will delete the object
hello.txt
.
2.4.4. PHP S3 Examples
You can use PHP scripts too for S3 access. Execute the steps mentioned below on the node used for accessing the Ceph Object Gateway server with PHP.
The examples given below are tested against php v5.4.16
and aws-sdk v2.8.24
. DO NOT use the latest version of aws-sdk
for php
as it requires php >= 5.5+
. php 5.5
is not available in the default repos of RHEL 7
. If you want to use php 5.5
, you will have to enable epel
and other third party repos. Also, the configuration options for php 5.5
and latest version of aws-sdk
are different.
Setup PHP/AWS SDK
Execute the following steps to set up PHP:
As
root
, installphp
:# yum install php
Install
aws-sdk
for php:Download the zip archive of
aws-sdk
for php and extract it.
Creating a connection
Create a project directory:
$ mkdir php_s3 $ cd php_s3
Copy the extracted
aws
directory to the project directory. For example:$ cp -r ~/Downloads/aws/ ~/php_s3/
Create the connection file:
$ vim conn.php
Paste the following contents in the
conn.php
file:<?php define('AWS_KEY', 'my_access_key'); define('AWS_SECRET_KEY', 'my_secret_key'); define('HOST', '<FQDN_of_gateway_node>'); define('PORT', '8080'); // require the AWS SDK for php library require '/path_to_aws/aws-autoloader.php'; use Aws\S3\S3Client; // Establish connection with host using S3 Client $client = S3Client::factory(array( 'base_url' => HOST, 'port' => PORT, 'key' => AWS_KEY, 'secret' => AWS_SECRET_KEY )); ?>
Replace
<FQDN_of_gateway_node>
with the FQDN of the gateway node. Replacemy-access-key
andmy-secret-key
with theaccess_key
andsecret_key
that were generated when you created theradosgw
user forS3
access as mentioned in the Red Hat Ceph Storage Object Gateway Guide. Also, replacepath_to_aws
with absolute path to the extractedaws
directory that you copied to thephp
project directory.An example connection file will look like the following:
<?php define('AWS_KEY', '{key}'); define('AWS_SECRET_KEY', '{secret}'); define('HOST', 'http://{hostname}'); // require the AWS SDK for php library require '/home/ceph/php_s3/aws/aws-autoloader.php'; use Aws\S3\S3Client; // Establish connection with host using S3 Client $client = S3Client::factory(array( 'base_url' => HOST, 'port' => PORT, 'key' => AWS_KEY, 'secret' => AWS_SECRET_KEY )); ?>
Save the file and exit the editor.
Run the file:
$ php -f conn.php | echo $?
If you have provided the values correctly in the file, the output of the command will be
0
.
Creating a bucket
Create a new file:
vim create_bucket.php
Paste the following contents into the file:
<?php include 'conn.php'; $client->createBucket(array('Bucket' => 'my-new-bucket3')); ?>
Save the file and exit the editor.
Run the file:
php -f create_bucket.php
Listing owned buckets
Create a new file:
vim list_owned_buckets.php
Paste the following content into the file:
<?php include 'conn.php'; $blist = $client->listBuckets(); echo " Buckets belonging to " . $blist['Owner']['ID'] . ":\n"; foreach ($blist['Buckets'] as $b) { echo "{$b['Name']}\t{$b['CreationDate']}\n"; } ?>
Save the file and exit the editor.
Run the file:
php -f list_owned_buckets.php
The output should look something like this:
my-new-bucket3 2016-01-21 10:33:19 UTC
Creating an object
Create a source file
hello.txt
:echo "Hello World!" > hello.txt
Create a new php file:
vim create_object.php
Paste the following contents into the file:
<?php include 'conn.php'; $key = 'hello.txt'; $source_file = './hello.txt'; $acl = 'private'; $bucket = 'my-new-bucket3'; $client->upload($bucket, $key, fopen($source_file, 'r'), $acl); ?>
Save the file and exit the editor.
Run the file:
php -f create_object.php
This will create the object
hello.txt
in bucketmy-new-bucket3
.
Listing a Bucket’s Content
Create a new file:
vim list_bucket_content.php
Paste the following content into the file:
<?php include 'conn.php'; $o_iter = $client->getIterator('ListObjects', array( 'Bucket' => 'my-new-bucket3' )); foreach ($o_iter as $o) { echo "{$o['Key']}\t{$o['Size']}\t{$o['LastModified']}\n"; } ?>
Save the file and exit the editor.
Run the file:
php -f list_bucket_content.php
The output will look something like this:
hello.txt 12 Fri, 22 Jan 2016 15:54:52 GMT
Deleting an empty bucket
Create a new file:
vim del_empty_bucket.php
Paste the following contents into the file:
<?php include 'conn.php'; $client->deleteBucket(array('Bucket' => 'my-new-bucket3')); ?>
Save the file and exit the editor.
Run the file:
php -f del_empty_bucket.php | echo $?
If the bucket is successfully deleted, the command will return
0
as output.NoteEdit the
create_bucket.php
file to create empty buckets likemy-new-bucket4
,my-new-bucket5
, and so on, and then edit thedel_empty_bucket.php
file accordingly before trying to delete empty buckets.
Deleting a non-empty bucket (forcefully)
Deleting a non-empty bucket is not currently supported in php 2
and newer versions of aws-sdk
.
Deleting an object
Create a new file:
vim delete_object.php
Paste the following contents into the file:
<?php include 'conn.php'; $client->deleteObject(array( 'Bucket' => 'my-new-bucket3', 'Key' => 'hello.txt', )); ?>
Save the file and exit the editor.
Run the file:
php -f delete_object.php
This will delete the object
hello.txt
.
2.4.5. Configuring and using STS Lite with Keystone (Technology Preview)
The Amazon Secure Token Service (STS) and S3 APIs co-exist in the same namespace. The STS options can be configured in conjunction with the Keystone options.
Both S3 and STS APIs can be accessed using the same endpoint in Ceph Object Gateway.
Prerequisites
- Red Hat Ceph Storage 3.2 or higher.
- A running Ceph Object Gateway.
- Installation of the Boto Python module, version 3 or higher.
Procedure
Open and edit the
group_vars/rgws.yml
file with the following options:rgw_sts_key = $STS_KEY_FOR_ENCRYPTING_SESSION_TOKEN rgw_s3_auth_use_sts = true
Rerun the Ansible playbook:
[user@admin ceph-ansible]$ ansible-playbook site.yml --limit rgws
Generate the EC2 credentials:
Example
$ openstack ec2 credentials create +------------+--------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------+ | access | b924dfc87d454d15896691182fdeb0ef | | links | {u'self': u'http://192.168.0.15/identity/v3/users/ | | | 40a7140e424f493d8165abc652dc731c/credentials/ | | | OS-EC2/b924dfc87d454d15896691182fdeb0ef'} | | project_id | c703801dccaf4a0aaa39bec8c481e25a | | secret | 6a2142613c504c42a94ba2b82147dc28 | | trust_id | None | | user_id | 40a7140e424f493d8165abc652dc731c | +------------+--------------------------------------------------------+
Use the generated credentials to get back a set of temporary security credentials using GetSessionToken API.
Example
import boto3 access_key = b924dfc87d454d15896691182fdeb0ef secret_key = 6a2142613c504c42a94ba2b82147dc28 client = boto3.client('sts', aws_access_key_id=access_key, aws_secret_access_key=secret_key, endpoint_url=https://www.example.com/rgw, region_name='', ) response = client.get_session_token( DurationSeconds=43200 )
Obtaining the temporary credentials can be used for making S3 calls:
Example
s3client = boto3.client('s3', aws_access_key_id = response['Credentials']['AccessKeyId'], aws_secret_access_key = response['Credentials']['SecretAccessKey'], aws_session_token = response['Credentials']['SessionToken'], endpoint_url=https://www.example.com/s3, region_name='') bucket = s3client.create_bucket(Bucket='my-new-shiny-bucket') response = s3client.list_buckets() for bucket in response["Buckets"]: print "{name}\t{created}".format( name = bucket['Name'], created = bucket['CreationDate'], )
Create a new S3Access role and configure a policy.
Assign a user with administrative CAPS:
radosgw-admin caps add --uid="$USER" --caps="roles=*"
Example
[user@client]$ radosgw-admin caps add --uid="gwadmin" --caps="roles=*"
Create the S3Access role:
radosgw-admin role create --role-name=$ROLE_NAME --path=$PATH --assume-role-policy-doc=$TRUST_POLICY_DOC
Example
[user@client]$ radosgw-admin role create --role-name=S3Access --path=/application_abc/component_xyz/ --assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"AWS\":\[\"arn:aws:iam:::user/TESTER\"\]\},\"Action\":\[\"sts:AssumeRole\"\]\}\]\}
Attach a permission policy to the S3Access role:
radosgw-admin role-policy put --role-name=$ROLE_NAME --policy-name=$POLICY_NAME --policy-doc=$PERMISSION_POLICY_DOC
Example
[user@client]$ radosgw-admin role-policy put --role-name=S3Access --policy-name=Policy --policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Action\":\[\"s3:*\"\],\"Resource\":\"arn:aws:s3:::example_bucket\"\}\]\}
-
Now another user can assume the role of the
gwadmin
user. For example, thegwuser
user can assume the permissions of thegwadmin
user. Make a note of the assuming user’s
access_key
andsecret_key
values.Example
[user@client]$ radosgw-admin user info --uid=gwuser | grep -A1 access_key
Use the AssumeRole API call, providing the
access_key
andsecret_key
values from the assuming user:Example
import boto3 access_key = 11BS02LGFB6AL6H1ADMW secret_key = vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY client = boto3.client('sts', aws_access_key_id=access_key, aws_secret_access_key=secret_key, endpoint_url=https://www.example.com/rgw, region_name='', ) response = client.assume_role( RoleArn='arn:aws:iam:::role/application_abc/component_xyz/S3Access', RoleSessionName='Bob', DurationSeconds=3600 )
ImportantThe AssumeRole API requires the S3Access role.
Additional Resources
- See the Test S3 Access section in the Red Hat Ceph Storage Object Gateway Guide for more information on installing the Boto Python module.
- See the Create a User section in the Red Hat Ceph Storage Object Gateway Guide for more information.
2.4.6. Working around the limitations of using STS Lite with Keystone (Technology Preview)
A limitation with Keystone is that it does not supports STS requests. Another limitation is the payload hash is not included with the request. To work around these two limitations the Boto authentication code must be modified.
Prerequisites
- Red Hat Ceph Storage 3.2 or higher.
- A running Ceph Object Gateway.
- Installation of Boto Python module, version 3 or higher.
Procedure
Open and edit Boto’s
auth.py
file.Add the following four lines to the code block:
class SigV4Auth(BaseSigner): """ Sign a request with Signature V4. """ REQUIRES_REGION = True def __init__(self, credentials, service_name, region_name): self.credentials = credentials # We initialize these value here so the unit tests can have # valid values. But these will get overriden in ``add_auth`` # later for real requests. self._region_name = region_name if service_name == 'sts': 1 self._service_name = 's3' 2 else: 3 self._service_name = service_name 4
Add the following two lines to the code block:
def _modify_request_before_signing(self, request): if 'Authorization' in request.headers: del request.headers['Authorization'] self._set_necessary_date_headers(request) if self.credentials.token: if 'X-Amz-Security-Token' in request.headers: del request.headers['X-Amz-Security-Token'] request.headers['X-Amz-Security-Token'] = self.credentials.token if not request.context.get('payload_signing_enabled', True): if 'X-Amz-Content-SHA256' in request.headers: del request.headers['X-Amz-Content-SHA256'] request.headers['X-Amz-Content-SHA256'] = UNSIGNED_PAYLOAD 1 else: 2 request.headers['X-Amz-Content-SHA256'] = self.payload(request)
Additional Resources
- See the Test S3 Access section in the Red Hat Ceph Storage Object Gateway Guide for more information on installing the Boto Python module.
2.5. Common Operations
2.5.1. Bucket and Host Name
There are two different modes of accessing the buckets. The first, and preferred method identifies the bucket as the top-level directory in the URI.
Example
GET /mybucket HTTP/1.1 Host: cname.domain.com
The second method identifies the bucket via a virtual bucket host name.
Example
GET / HTTP/1.1 Host: mybucket.cname.domain.com
Red Hat prefers the first method, because the second method requires expensive domain certification and DNS wild cards.
2.5.2. Common Request Headers
The following table lists the valid common request headers and their descriptions.
Request Header | Description |
---|---|
| Length of the request body. |
| Request time and date (in UTC). |
| The name of the host server. |
| Authorization token. |
2.5.3. Common Response Status
The following table lists the valid common HTTP response status and its corresponding code.
HTTP Status | Response Code |
---|---|
| Continue |
| Success |
| Created |
| Accepted |
| NoContent |
| Partial content |
| NotModified |
| InvalidArgument |
| InvalidDigest |
| BadDigest |
| InvalidBucketName |
| InvalidObjectName |
| UnresolvableGrantByEmailAddress |
| InvalidPart |
| InvalidPartOrder |
| RequestTimeout |
| EntityTooLarge |
| AccessDenied |
| UserSuspended |
| RequestTimeTooSkewed |
| NoSuchKey |
| NoSuchBucket |
| NoSuchUpload |
| MethodNotAllowed |
| RequestTimeout |
| BucketAlreadyExists |
| BucketNotEmpty |
| MissingContentLength |
| PreconditionFailed |
| InvalidRange |
| UnprocessableEntity |
| InternalError |
2.6. Service Operations
2.6.1. List Buckets
GET /
returns a list of buckets created by the user making the request. GET /
only returns buckets created by an authenticated user. You cannot make an anonymous request.
Syntax
GET / HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Name | Type | Description |
---|---|---|
| Container | Container for list of buckets. |
| Container | Container for bucket information. |
| String | Bucket name. |
| Date | UTC time when the bucket was created. |
| Container | A container for the result. |
| Container |
A container for the bucket owner’s |
| String | The bucket owner’s ID. |
| String | The bucket owner’s display name. |
2.7. Bucket Operations
2.7.1. Bucket Operations with Multi Tenancy
When a client application accesses buckets, it always operates with credentials of a particular user. In Red Hat Ceph Storage 3, every user belongs to a tenant. See Multi Tenancy for additional details. Consequently, every bucket operation has an implicit tenant in its context if no tenant is specified explicitly. Thus multi tenancy is completely backward compatible with previous releases, as long as the referred buckets and referring user belong to the same tenant.
Extensions employed to specify an explicit tenant differ according to the protocol and authentication system used.
In the following example, a colon character separates tenant and bucket. Thus a sample URL would be:
https://rgw.domain.com/tenant:bucket
By contrast, a simple Python example separates the tenant and bucket in the bucket method itself:
from boto.s3.connection import S3Connection, OrdinaryCallingFormat c = S3Connection( aws_access_key_id="TESTER", aws_secret_access_key="test123", host="rgw.domain.com", calling_format = OrdinaryCallingFormat() ) bucket = c.get_bucket("tenant:bucket")
It’s not possible to use S3-style subdomains using multi-tenancy, since host names cannot contain colons or any other separators that are not already valid in bucket names. Using a period creates an ambiguous syntax. Therefore, the bucket-in-URL-path
format has to be used with multi-tenancy.
2.7.2. Bucket Lifecycle
You can use a bucket lifecycle configuration to manage your objects so they are stored effectively throughout their lifetime. The S3 API in the Ceph Object Gateway supports a subset of the AWS bucket lifecycle actions:
-
Expiration
: This defines the lifespan of objects within a bucket. It takes the number of days the object should live or an expiration date, at which point Ceph Object Gateway will delete the object. If the bucket doesn’t enable versioning, Ceph Object Gateway will delete the object permanently. If the bucket enables versioning, Ceph Object Gateway will create a delete marker for the current version, and then delete the current version. -
NoncurrentVersionExpiration
: This defines the lifespan of non-current object versions within a bucket. To use this feature, the bucket must enable versioning. It takes the number of days a non-current object should live, at which point Ceph Object Gateway will delete the non-current object. -
AbortIncompleteMultipartUpload
: This defines the number of days an incomplete multipart upload should live before it is aborted.
The lifecycle configuration contains one or more rules using the <Rule>
element. For example:
<LifecycleConfiguration> <Rule> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>10</Days> </Expiration> </Rule> </LifecycleConfiguration>
A lifecycle rule can apply to all or a subset of objects in a bucket based on the <Filter>
element that you specify in the lifecycle rule. You can specify a filter several ways:
- Key prefixes
- Object tags
- Both key prefix and one or more object tags
Key prefixes
You can apply a lifecycle rule to a subset of objects based on the key name prefix. For example, specifying <keypre/>
would apply to objects that begin with keypre/
:
<LifecycleConfiguration> <Rule> <Status>Enabled</Status> <Filter> <Prefix>keypre/</Prefix> </Filter> </Rule> </LifecycleConfiguration>
You can also apply different lifecycle rules to objects with different key prefixes:
<LifecycleConfiguration> <Rule> <Status>Enabled</Status> <Filter> <Prefix>keypre/</Prefix> </Filter> </Rule> <Rule> <Filter> <Prefix>mypre/</Prefix> </Filter> </Rule> </LifecycleConfiguration>
Object tags
You can apply a lifecycle rule to only objects with a specific tag using the <Key>
and <Value>
elements:
<LifecycleConfiguration> <Rule> <Status>Enabled</Status> <Filter> <Tag> <Key>key</Key> <Value>value</Value> </Tag> </Filter> </Rule> </LifecycleConfiguration>
Both prefix and one or more tags
In a lifecycle rule, you can specify a filter based on both the key prefix and one or more tags. They must be wrapped in the <And>
element. A filter can have only one prefix, and zero or more tags:
<LifecycleConfiguration> <Rule> <Status>Enabled</Status> <Filter> <And> <Prefix>key-prefix</Prefix> <Tag> <Key>key1</Key> <Value>value1</Value> </Tag> <Tag> <Key>key2</Key> <Value>value2</Value> </Tag> ... </And> </Filter> <Status>Enabled</Status> </Rule> </LifecycleConfiguration>
For additional details, see:
2.7.3. Head Bucket
Calls HEAD on a bucket to determine if it exists and if the caller has access permissions. Returns 200 OK
if the bucket exists and the caller has permissions; 404 Not Found
if the bucket does not exist; and, 403 Forbidden
if the bucket exists but the caller does not have access permissions.
Syntax
HEAD /<bucket> HTTP/1.1 Host: cname.domain.com Date: date Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.7.4. PUT Bucket
Creates a new bucket. To create a bucket, you must have a user ID and a valid AWS Access Key ID to authenticate requests. You can not create buckets as an anonymous user.
Constraints
In general, bucket names should follow domain name constraints.
- Bucket names must be unique.
- Bucket names must begin and end with a lowercase letter.
- Bucket names can contain a dash (-).
Syntax
PUT /<bucket> HTTP/1.1 Host: cname.domain.com x-amz-acl: public-read-write Authorization: AWS <access_key>:<hash_of_header_and_secret>
Name | Description | Valid Values | Required |
---|---|---|---|
| Canned ACLs. |
| No |
HTTP Response
If the bucket name is unique, within constraints and unused, the operation will succeed. If a bucket with the same name already exists and the user is the bucket owner, the operation will succeed. If the bucket name is already in use, the operation will fail.
HTTP Status | Status Code | Description |
---|---|---|
| BucketAlreadyExists | Bucket already exists under different user’s ownership. |
2.7.5. Put Bucket Lifecycle
To create or replace a bucket lifecycle, use PUT
and specify a destination bucket and a lifecycle configuration. The Ceph Object Gateway only supports a subset of the S3 lifecycle functionality. See S3 API Bucket Lifecycle for details.
Syntax
PUT /<bucket>?lifecycle HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret> <LifecycleConfiguration> <Rule> <Expiration> <Days>10</Days> </Expiration> </Rule> ... <Rule> </Rule> </LifecycleConfiguration>
Name | Description | Valid Values | Required |
---|---|---|---|
content-md5 | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No |
See also Common Request Headers
2.7.6. DELETE Bucket
Deletes a bucket. You can reuse bucket names following a successful bucket removal.
Syntax
DELETE /<bucket> HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
HTTP Status | Status Code | Description |
---|---|---|
| No Content | Bucket removed. |
2.7.7. Delete Bucket Lifecycle
To delete a bucket lifecycle, use DELETE
and specify a destination bucket.
Syntax
DELETE /<bucket>?lifecycle HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Request Headers
The request does not contain any special elements.
Response
The response returns common response status.
See Common Response Status for details.
2.7.8. GET Bucket
Returns a list of bucket objects.
Syntax
GET /<bucket>?max-keys=25 HTTP/1.1 Host: cname.domain.com
Name | Type | Description |
---|---|---|
| String | Only returns objects that contain the specified prefix. |
| String | The delimiter between the prefix and the rest of the object name. |
| String | A beginning index for the list of objects returned. |
| Integer | The maximum number of keys to return. Default is 1000. |
HTTP Status | Status Code | Description |
---|---|---|
| OK | Buckets retrieved |
GET /<bucket>
returns a container for buckets with the following fields:
Name | Type | Description |
---|---|---|
| Entity | The container for the list of objects. |
| String | The name of the bucket whose contents will be returned. |
| String | A prefix for the object keys. |
| String | A beginning index for the list of objects returned. |
| Integer | The maximum number of keys returned. |
| String |
If set, objects with the same prefix will appear in the |
| Boolean |
If |
| Container | If multiple objects contain the same prefix, they will appear in this list. |
The ListBucketResult
contains objects, where each object is within a Contents
container.
Name | Type | Description |
---|---|---|
| Object | A container for the object. |
| String | The object’s key. |
| Date | The object’s last-modified date/time. |
| String | An MD-5 hash of the object. (entity tag) |
| Integer | The object’s size. |
| String |
Should always return |
2.7.9. Get Bucket Lifecycle
To get a bucket lifecycle, use GET
and specify a destination bucket.
Syntax
GET /<bucket>?lifecycle HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Request Headers
Response
The response contains the bucket lifecycle and its elements.
2.7.10. Get Bucket Location
Retrieves the bucket’s zone group. The user needs to be the bucket owner to call this. A bucket can be constrained to a zone group by providing LocationConstraint
during a PUT request.
Add the location
subresource to bucket resource as shown below.
Syntax
GET /<bucket>?location HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Name | Type | Description |
---|---|---|
| String | The zone group where bucket resides, empty string for defult zone group |
2.7.11. Get Bucket Versioning
Retrieves the versioning state of a bucket. The user needs to be the bucket owner to call this.
Add the versioning
subresource to bucket resource as shown below.
Syntax
GET /<bucket>?versioning HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.7.12. PUT Bucket Versioning
This subresource set the versioning state of an existing bucket. The user needs to be the bucket owner to set the versioning state. If the versioning state has never been set on a bucket, then it has no versioning state. Doing a GET versioning request does not return a versioning state value.
Setting the bucket versioning state:
Enabled
: Enables versioning for the objects in the bucket. All objects added to the bucket receive a unique version ID. Suspended
: Disables versioning for the objects in the bucket. All objects added to the bucket receive the version ID null.
Syntax
PUT /<bucket>?versioning HTTP/1.1
Name | Type | Description |
---|---|---|
| container | A container for the request. |
| String | Sets the versioning state of the bucket. Valid Values: Suspended/Enabled |
2.7.13. Get Bucket ACLs
Retrieves the bucket access control list. The user needs to be the bucket owner or to have been granted READ_ACP
permission on the bucket.
Add the acl
subresource to the bucket request as shown below.
Syntax
GET /<bucket>?acl HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Name | Type | Description |
---|---|---|
| Container | A container for the response. |
| Container | A container for the ACL information. |
| Container |
A container for the bucket owner’s |
| String | The bucket owner’s ID. |
| String | The bucket owner’s display name. |
| Container |
A container for |
| Container |
A container for the |
| String |
The permission given to the |
2.7.14. PUT Bucket ACLs
Sets an access control to an existing bucket. The user needs to be the bucket owner or to have been granted WRITE_ACP
permission on the bucket.
Add the acl
subresource to the bucket request as shown below.
Syntax
PUT /<bucket>?acl HTTP/1.1
Name | Type | Description |
---|---|---|
| Container | A container for the request. |
| Container | A container for the ACL information. |
| Container |
A container for the bucket owner’s |
| String | The bucket owner’s ID. |
| String | The bucket owner’s display name. |
| Container |
A container for |
| Container |
A container for the |
| String |
The permission given to the |
2.7.15. GET Bucket cors
Retrieves the cors configuration information set for the bucket. The user needs to be the bucket owner or to have been granted READ_ACP
permission on the bucket.
Add the cors
subresource to the bucket request as shown below.
Syntax
GET /<bucket>?cors HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.7.16. PUT Bucket cors
Sets the cors configuration for the bucket. The user needs to be the bucket owner or to have been granted READ_ACP
permission on the bucket.
Add the cors
subresource to the bucket request as shown below.
Syntax
PUT /<bucket>?cors HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.7.17. DELETE Bucket cors
Deletes the cors configuration information set for the bucket. The user needs to be the bucket owner or to have been granted READ_ACP
permission on the bucket.
Add the cors
subresource to the bucket request as shown below.
Syntax
DELETE /<bucket>?cors HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.7.18. List Bucket Object Versions
Returns a list of metadata about all the version of objects within a bucket. Requires READ access to the bucket.
Add the versions
subresource to the bucket request as shown below.
Syntax
GET /<bucket>?versions HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
You can specify parameters for GET /<bucket>?versions
, but none of them are required.
Name | Type | Description |
---|---|---|
| String | Returns in-progress uploads whose keys contains the specified prefix. |
| String | The delimiter between the prefix and the rest of the object name. |
| String | The beginning marker for the list of uploads. |
| Integer | The maximum number of in-progress uploads. The default is 1000. |
| String | Specifies the object version to begin the list. |
Name | Type | Description |
---|---|---|
| String |
The key marker specified by the |
| String |
The key marker to use in a subsequent request if |
| String |
The upload ID marker to use in a subsequent request if |
| Boolean |
If |
| Integer | The size of the uploaded part. |
| String | The owners’s display name. |
| String | The owners’s ID. |
| Container |
A container for the |
| String |
The method used to store the resulting object. |
| Container | Container for the version information. |
| String | Version ID of an object. |
| String | The last version of the key in a truncated response. |
2.7.19. List Bucket Multipart Uploads
GET /?uploads
returns a list of the current in-progress multipart uploads, that is, the application initiates a multipart upload, but the service hasn’t completed all the uploads yet.
Syntax
GET /<bucket>?uploads HTTP/1.1
You can specify parameters for GET /<bucket>?uploads
, but none of them are required.
Name | Type | Description |
---|---|---|
| String | Returns in-progress uploads whose keys contains the specified prefix. |
| String | The delimiter between the prefix and the rest of the object name. |
| String | The beginning marker for the list of uploads. |
| Integer | The maximum number of in-progress uploads. The default is 1000. |
| Integer | The maximum number of multipart uploads. The range from 1-1000. The default is 1000. |
| String |
Ignored if |
Name | Type | Description |
---|---|---|
| Container | A container for the results. |
| String |
The prefix specified by the |
| String | The bucket that will receive the bucket contents. |
| String |
The key marker specified by the |
| String |
The marker specified by the |
| String |
The key marker to use in a subsequent request if |
| String |
The upload ID marker to use in a subsequent request if |
| Integer |
The max uploads specified by the |
| String |
If set, objects with the same prefix will appear in the |
| Boolean |
If |
| Container |
A container for |
| String | The key of the object once the multipart upload is complete. |
| String |
The |
| Container |
Contains the |
| String | The initiator’s display name. |
| String | The initiator’s ID. |
| Container |
A container for the |
| String |
The method used to store the resulting object. |
| Date | The date and time the user initiated the upload. |
| Container | If multiple objects contain the same prefix, they will appear in this list. |
| String |
The substring of the key after the prefix as defined by the |
2.7.20. PUT Bucket Request Payment
Uses the requestPayment
subresource to set the request payment configuration of a bucket. By default, the bucket owner pays for downloads from the bucket. This configuration parameter enables the bucket owner to specify that the person requesting the download will be charged for the request and the data download from the bucket.
Add the requestPayment
subresource to the bucket request as shown below.
Syntax
PUT /<bucket>?requestPayment HTTP/1.1 Host: cname.domain.com
Name | Type | Description |
---|---|---|
| Enum | Specifies who pays for the download and request fees. |
| Container |
A container for |
2.7.21. GET Bucket Request Payment
Uses the requestPayment
subresource to return the request payment configuration of a bucket. The user needs to be the bucket owner or to have been granted READ_ACP
permission on the bucket.
Add the requestPayment
subresource to the bucket request as shown below.
Syntax
GET /<bucket>?requestPayment HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
2.8. Object Operations
2.8.1. PUT Object
Adds an object to a bucket. You must have write permissions on the bucket to perform this operation.
Syntax
PUT /<bucket>/<object> HTTP/1.1
Name | Description | Valid Values | Required |
---|---|---|---|
content-md5 | A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No |
content-type | A standard MIME type. |
Any MIME type. Default: | No |
x-amz-meta-<…> | User metadata. Stored with the object. | A string up to 8kb. No defaults. | No |
x-amz-acl | A canned ACL. |
| No |
Name | Description |
---|---|
x-amz-version-id | Returns the version ID or null. |
2.8.2. Copy Object
To copy an object, use PUT
and specify a destination bucket and the object name.
Syntax
PUT /<dest_bucket>/<dest_object> HTTP/1.1 x-amz-copy-source: <source_bucket>/<source_object>
Name | Description | Valid Values | Required |
---|---|---|---|
x-amz-copy-source | The source bucket name + object name. | <bucket>/<object> | Yes |
x-amz-acl | A canned ACL. |
| No |
x-amz-copy-if-modified-since | Copies only if modified since the timestamp. | Timestamp | No |
x-amz-copy-if-unmodified-since | Copies only if unmodified since the timestamp. | Timestamp | No |
x-amz-copy-if-match | Copies only if object ETag matches ETag. | Entity Tag | No |
x-amz-copy-if-none-match | Copies only if object ETag doesn’t match. | Entity Tag | No |
Name | Type | Description |
---|---|---|
CopyObjectResult | Container | A container for the response elements. |
LastModified | Date | The last modified date of the source object. |
Etag | String | The ETag of the new object. |
2.8.3. POST Object
Adds an object to a bucket using HTML forms. You must have write permissions on the bucket to perform this operation.
Syntax
POST /<bucket>/<object> HTTP/1.1
2.8.4. OPTIONS Object
A preflight request to determine if an actual request can be sent with the specific origin, HTTP method, and headers.
Syntax
OPTIONS /<object> HTTP/1.1
2.8.5. Delete Multiple Objects
Deletes multiple objects from a bucket.
Syntax
POST /<bucket>/<object>?delete HTTP/1.1
2.8.6. Remove Object
Removes an object. Requires WRITE permission set on the containing bucket.
Deletes an object. If object versioning is on, it creates a marker.
Syntax
DELETE /<bucket>/<object> HTTP/1.1
To delete an object when versioning is on, you must specify the versionId
subresource and the version of the object to delete.
DELETE /<bucket>/<object>?versionId=<versionID> HTTP/1.1
2.8.7. Get Object
Retrieves an object from a bucket:
Syntax
GET /<bucket>/<object> HTTP/1.1
Add the versionId
subresource to retrieve a particular version of the object:
Syntax
GET /<bucket>/<object>?versionId=<versionID> HTTP/1.1
Name | Description | Valid Values | Required |
---|---|---|---|
range | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No |
if-modified-since | Gets only if modified since the timestamp. | Timestamp | No |
if-unmodified-since | Gets only if not modified since the timestamp. | Timestamp | No |
if-match | Gets only if object ETag matches ETag. | Entity Tag | No |
if-none-match | Gets only if object ETag matches ETag. | Entity Tag | No |
Name | Description |
---|---|
Content-Range | Data range, will only be returned if the range header field was specified in the request |
x-amz-version-id | Returns the version ID or null. |
2.8.8. Get Object Information
Returns information about an object. This request will return the same header information as with the Get Object request, but will include the metadata only, not the object data payload.
Retrieves the current version of the object:
Syntax
HEAD /<bucket>/<object> HTTP/1.1
Add the versionId
subresource to retrieve info for a particular version:
Syntax
HEAD /<bucket>/<object>?versionId=<versionID> HTTP/1.1
Name | Description | Valid Values | Required |
---|---|---|---|
range | The range of the object to retrieve. | Range: bytes=beginbyte-endbyte | No |
if-modified-since | Gets only if modified since the timestamp. | Timestamp | No |
if-unmodified-since | Gets only if not modified since the timestamp. | Timestamp | No |
if-match | Gets only if object ETag matches ETag. | Entity Tag | No |
if-none-match | Gets only if object ETag matches ETag. | Entity Tag | No |
Name | Description |
---|---|
x-amz-version-id | Returns the version ID or null. |
2.8.9. Get Object ACL
Returns the ACL for the current version of the object:
Syntax
GET /<bucket>/<object>?acl HTTP/1.1
Add the versionId
subresource to retrieve the ACL for a particular version:
Syntax
GET /<bucket>/<object>versionId=<versionID>&acl HTTP/1.1
Name | Description |
---|---|
x-amz-version-id | Returns the version ID or null. |
Name | Type | Description |
---|---|---|
| Container | A container for the response. |
| Container | A container for the ACL information. |
| Container |
A container for the object owner’s |
| String | The object owner’s ID. |
| String | The object owner’s display name. |
| Container |
A container for |
| Container |
A container for the |
| String |
The permission given to the |
2.8.10. Set Object ACL
Sets an object ACL for the current version of the object.
Syntax
PUT /<bucket>/<object>?acl
Name | Type | Description |
---|---|---|
| Container | A container for the response. |
| Container | A container for the ACL information. |
| Container |
A container for the object owner’s |
| String | The object owner’s ID. |
| String | The object owner’s display name. |
| Container |
A container for |
| Container |
A container for the |
| String |
The permission given to the |
2.8.11. Initiate Multipart Upload
Initiates a multi-part upload process. Returns a UploadId
, which you can specify when adding additional parts, listing parts, and completing or abandoning a multi-part upload.
Syntax
POST /<bucket>/<object>?uploads
Name | Description | Valid Values | Required |
---|---|---|---|
| A base64 encoded MD-5 hash of the message. | A string. No defaults or constraints. | No |
| A standard MIME type. |
Any MIME type. Default: | No |
| User metadata. Stored with the object. | A string up to 8kb. No defaults. | No |
| A canned ACL. |
| No |
Name | Type | Description |
---|---|---|
| Container | A container for the results. |
| String | The bucket that will receive the object contents. |
| String |
The key specified by the |
| String |
The ID specified by the |
2.8.12. Multipart Upload Part
Adds a part to a multi-part upload.
Specify the uploadId
subresource and the upload ID to add a part to a multi-part upload:
Syntax
PUT /<bucket>/<object>?partNumber=&uploadId=<upload_id> HTTP/1.1
The following HTTP response might be returned:
HTTP Status | Status Code | Description |
---|---|---|
| NoSuchUpload | Specified upload-id does not match any initiated upload on this object |
2.8.13. List Multipart Upload Parts
Specify the uploadId
subresource and the upload ID to list the parts of a multi-part upload:
Syntax
GET /<bucket>/<object>?uploadId=<upload-id> HTTP/1.1
Name | Type | Description |
---|---|---|
| Container | A container for the results. |
| String | The bucket that will receive the object contents. |
| String |
The key specified by the |
| String |
The ID specified by the |
| Container |
Contains the |
| String | The initiator’s ID. |
| String | The initiator’s display name. |
| Container |
A container for the |
| String |
The method used to store the resulting object. |
| String |
The part marker to use in a subsequent request if |
| String |
The next part marker to use in a subsequent request if |
| Integer |
The max parts allowed in the response as specified by the |
| Boolean |
If |
| Container |
A container for |
| Integer | The identification number of the part. |
| String | The part’s entity tag. |
| Integer | The size of the uploaded part. |
2.8.14. Complete Multipart Upload
Assembles uploaded parts and creates a new object, thereby completing a multipart upload.
Specify the uploadId
subresource and the upload ID to complete a multi-part upload:
Syntax
POST /<bucket>/<object>?uploadId= HTTP/1.1
Name | Type | Description | Required |
---|---|---|---|
| Container | A container consisting of one or more parts. | Yes |
| Container |
A container for the | Yes |
| Integer | The identifier of the part. | Yes |
| String | The part’s entity tag. | Yes |
Name | Type | Description |
---|---|---|
| Container | A container for the response. |
| URI | The resource identifier (path) of the new object. |
| String | The name of the bucket that contains the new object. |
| String | The object’s key. |
| String | The entity tag of the new object. |
2.8.15. Abort Multipart Upload
Aborts a multipart upload.
Specify the uploadId
subresource and the upload ID to abort a multi-part upload:
Syntax
DELETE /<bucket>/<object>?uploadId=<upload_id> HTTP/1.1
2.8.16. Copy Multipart Upload
Uploads a part by copying data from an existing object as data source.
Specify the uploadId
subresource and the upload ID to perform a multi-part upload copy:
Syntax
PUT /<bucket>/<object>?partNumber=PartNumber&uploadId=UploadId HTTP/1.1 Host: cname.domain.com Authorization: AWS <access_key>:<hash_of_header_and_secret>
Name | Description | Valid Values | Required |
---|---|---|---|
| The source bucket name and object name. | <bucket>/<object> | Yes |
| The range of bytes to copy from the source object. |
Range: | No |
Name | Type | Description |
---|---|---|
| Container | A container for all response elements. |
| String | Returns the ETag of the new part. |
| String | Returns the date the part was last modified. |
For more information about this feature, see the Amazon S3 site.
2.9. Hadoop S3A Interoperability
For data analytics applications that require Hadoop Distributed File System (HDFS) access, the Ceph Object Gateway can be accessed using the Apache S3A connector for Hadoop. The S3A connector is an open source tool that presents S3 compatible object storage as an HDFS file system with HDFS file system read and write semantics to the applications while data is stored in the Ceph Object Gateway.
Ceph Object Gateway is fully compatible with the S3A connector that ships with Hadoop 2.7.3.
2.10. S3 Limitations
The following limitations should be used with caution. There are implications related to your hardware selections, so you should always discuss these requirements with your Red Hat account team.
-
Maximum object size when using Amazon S3: Individual Amazon S3 objects can range in size from a minimum of 0B to a maximum of 5TB. The largest object that can be uploaded in a single
PUT
is 5GB. For objects larger than 100MB, you should consider using the Multipart Upload capability. - Maximum metadata size when using Amazon S3: There is no defined limit on the total size of user metadata that can be applied to an object, but a single HTTP request is limited to 16,000.
- The amount of data overhead Red Hat Ceph Storage produces to store S3 objects and metadata: The estimate here is 200-300 bytes plus the length of the object name. Versioned objects consume additional space proportional to the number of versions. Also, transient overhead is produced during multi-part upload and other transactional updates, but these overheads are recovered during garbage collection.