Object Gateway Configuration and Administration Guide

Procedure

1. Open for editing the Ceph configuration file, by default /etc/ceph/ceph.conf, and enable the Vault as the encryption key store:

   rgw_crypt_s3_kms_backend = vault

2. Under the [client.radosgw.INSTANCE_NAME] section, choose a Vault authentication method, either Token or the Vault agent.

   1. If using Token, then add the following lines:

      rgw_crypt_vault_auth = token
      rgw_crypt_vault_token_file = /etc/ceph/vault.token
      rgw_crypt_vault_addr = http://VAULT_SERVER:8200

   2. If using the Vault agent, then add the following lines:

      rgw_crypt_vault_auth = agent
      rgw_crypt_vault_addr = http://VAULT_SERVER:8100

3. Under the [client.radosgw.INSTANCE_NAME] section, choose a Vault secret engine, either Key/Value or Transit.

   1. If using Key/Value, then add the following line:

      rgw_crypt_vault_secret_engine = kv

   2. If using Transit, then add the following line:

      rgw_crypt_vault_secret_engine = transit

4. Optionally, Under the [client.radosgw.INSTANCE_NAME] section, you can set the Vault namespace where the encryption keys will be retrieved:

   rgw_crypt_vault_namespace = NAME_OF_THE_NAMESPACE

5. Restrict where the Ceph Object Gateway retrieves the encryption keys from the Vault by setting a path prefix:

   Example

   rgw_crypt_vault_prefix = /v1/secret/data

   1. For exportable Transit keys, set the prefix path as follows:

      rgw_crypt_vault_prefix = /v1/transit/export/encryption-key

      Assuming the domain name of the Vault server is vault-server, the Ceph Object Gateway will fetch encrypted transit keys from the following URL:

      Example

      http://vault-server:8200/v1/transit/export/encryption-key

6. Save the changes to the Ceph configuration file.

Procedure

1. Enable the Key/Value version 2 secret engine:

   [root@vault ~]# vault secrets enable kv-v2

2. Create a new key:

   Syntax

   vault kv put secret/PROJECT_NAME/BUCKET_NAME key=$(openssl rand -base64 32)

   Example

   [root@vault ~]# vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
   
   ====== Metadata ======
   Key              Value
   ---              -----
   created_time     2020-02-21T17:01:09.095824999Z
   deletion_time    n/a
   destroyed        false
   version          1

Procedure

1. Enable the Transit secret engine:

   [root@vault ~]# vault secrets enable transit

2. Create a new exportable key:

   Syntax

   vault write -f transit/keys/BUCKET_NAME exportable=true

   Example

   [root@vault ~]# vault write -f transit/keys/mybucketkey exportable=true

   Note

   By default the above command creates a aes256-gcm96 type key.

3. Verify the creation of the key:

   Syntax

   vault read transit/export/encryption-key/BUCKET_NAME/VERSION_NUMBER

   Example

   [root@vault ~]# vault read transit/export/encryption-key/mybucketkey/1
   
   Key     Value
   ---     -----
   keys    map[1:-gbTI9lNpqv/V/2lDcmH2Nq1xKn6FPDWarCmFM2aNsQ=]
   name    mybucketkey
   type    aes256-gcm96

   Note

   Providing the full key path, including the key version is required.

Procedure

1. Upload an object using the AWS command-line client:

   Example

   [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey

   Note

   The key fetching URL used in the example is: http://vault-server:8200/v1/secret/data/myproject/mybucketkey

Procedure

1. Create an S3 user:

   radosgw-admin user create --uid=name --display-name="First User"

   Replace name with the name of the S3 user, for example:

   [root@master-zone]# radosgw-admin user create --uid="testuser" --display-name="First User"
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "testuser",
               "access_key": "CEP28KDIQXBKU4M15PDC",
               "secret_key": "MARoio8HFc8JxhEilES3dKFVj8tV3NOOYymihTLO"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Verify the output to ensure that the values of access_key and secret_key do not include a JSON escape character (\). These values are needed for access validation, but certain clients cannot handle if the values include JSON escape character. To fix this problem, perform one of the following actions:

   • Remove the JSON escape character.
   • Encapsulate the string in quotes.
   • Regenerate the key and ensure that is does not include a JSON escape character.
   • Specify the key and secret manually.

   Do not remove the forward slash / because it is a valid character.

Procedure

1. Create the Swift user:

   Syntax

   radosgw-admin subuser create --uid=NAME --subuser=NAME:swift --access=full

   Replace NAME with the Swift user name, for example:

   Example

   [root@rgw]# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [
           {
               "id": "testuser:swift",
               "permissions": "full-control"
           }
       ],
       "keys": [
           {
               "user": "testuser",
               "access_key": "O8JDE41XMI74O185EHKD",
               "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
           }
       ],
       "swift_keys": [
           {
               "user": "testuser:swift",
               "secret_key": "13TLtdEW7bCqgttQgPzxFxziu0AgabtOc6vM8DLA"
           }
       ],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Create the secret key:

   Syntax

   radosgw-admin key create --subuser=NAME:swift --key-type=swift --gen-secret

   Replace NAME with the Swift user name, for example:

   Example

   [root@rgw]# radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [
           {
               "id": "testuser:swift",
               "permissions": "full-control"
           }
       ],
       "keys": [
           {
               "user": "testuser",
               "access_key": "O8JDE41XMI74O185EHKD",
               "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
           }
       ],
       "swift_keys": [
           {
               "user": "testuser:swift",
               "secret_key": "a4ioT4jEP653CDcdU8p4OuhruwABBRZmyNUbnSSt"
           }
       ],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

Procedure

1. Enable the common repository for Red Hat Enterprise Linux 7 and the High Availability repository for Red Hat Enterprise Linux 8:

   Red Hat Enterprise Linux 7

   # subscription-manager repos --enable=rhel-7-server-rh-common-rpms

   Red Hat Enterprise Linux 8

   # subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

2. Install the python-boto package.

   Red Hat Enterprise Linux 7

   # yum install python-boto

   Red Hat Enterprise Linux 8

   # dnf install python3-boto3

3. Create the Python script:

   vi s3test.py

4. Add the following contents to the file:

   Red Hat Enterprise Linux 7

   import boto
   import boto.s3.connection
   
   access_key = 'ACCESS'
   secret_key = 'SECRET'
   
   boto.config.add_section('s3')
   
   conn = boto.connect_s3(
           aws_access_key_id = access_key,
           aws_secret_access_key = secret_key,
           host = 's3.ZONE.hostname',
           port = PORT,
           is_secure=False,
           calling_format = boto.s3.connection.OrdinaryCallingFormat(),
           )
   
   bucket = conn.create_bucket('my-new-bucket')
   for bucket in conn.get_all_buckets():
   	print "{name}\t{created}".format(
   		name = bucket.name,
   		created = bucket.creation_date,
   )

   Red Hat Enterprise Linux 8

   import boto3
   
   endpoint = "" # enter the endpoint URL along with the port "http://URL:_PORT_"
   
   access_key = 'ACCESS'
   secret_key = 'SECRET'
   
   s3 = boto3.client(
           's3',
           endpoint_url=endpoint,
           aws_access_key_id=access_key,
           aws_secret_access_key=secret_key
           )
   
   s3.create_bucket(Bucket='my-new-bucket')
   
   response = s3.list_buckets()
   for bucket in response['Buckets']:
       print("{name}\t{created}".format(
   		name = bucket['Name'],
   		created = bucket['CreationDate']
   ))

   1. Replace ZONE with the zone name of the host where you have configured the gateway service. That is, the gateway host. Ensure that the host`setting resolves with DNS. Replace `PORT with the port number of the gateway.
   2. Replace ACCESS and SECRET with the access_key and secret_key values from the Create an S3 User section in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

5. Run the script:

   Red Hat Enterprise Linux 7

   python s3test.py

   Red Hat Enterprise Linux 8

   python3 s3test.py

   Example output:

   my-new-bucket 2021-08-16T17:09:10.000Z

1. Install Red Hat Enterprise Linux 7.

2. Register the nodes.

   [root@haproxy]# subscription-manager register

3. Enable the RHEL server repository.

   [root@haproxy]# subscription-manager repos --enable=rhel-7-server-rpms

4. Update the server.

   [root@haproxy]# yum update -y

5. Install admin tools (e.g., wget, vim, etc.) as needed.

6. Open port 80.

   [root@haproxy]# firewall-cmd --zone=public --add-port 80/tcp --permanent
   [root@haproxy]# firewall-cmd --reload

7. For HTTPS, open port 443.

   [root@haproxy]# firewall-cmd --zone=public --add-port 443/tcp --permanent
   [root@haproxy]# firewall-cmd --reload

8. Connect to the required port.

   [root@haproxy]# semanage port -m -t http_cache_port_t -p tcp 8081

Procedure

1. Install keepalived:

   [root@haproxy]# yum install -y keepalived

2. Configure keepalived on both HAProxy nodes:

   [root@haproxy]# vim /etc/keepalived/keepalived.conf

   In the configuration file, there is a script to check the haproxy processes:

   vrrp_script chk_haproxy {
     script "killall -0 haproxy" # check the haproxy process
     interval 2 # every 2 seconds
     weight 2 # add 2 points if OK
   }

   Next, the instance on the master and backup load balancers uses eno1 as the network interface. It also assigns a virtual IP address, that is, 192.168.1.20.

   Master load balancer node

   vrrp_instance RGW {
       state MASTER # might not be necessary. This is on the Master LB node.
       @main interface eno1
       priority 100
       advert_int 1
       interface eno1
       virtual_router_id 50
       @main unicast_src_ip 10.8.128.43 80
       unicast_peer {
              10.8.128.53
              }
       authentication {
           auth_type PASS
           auth_pass 1111
       }
       virtual_ipaddress {
           192.168.1.20
       }
       track_script {
         chk_haproxy
       }
   }
   virtual_server 192.168.1.20 80 eno1 { #populate correct interface
       delay_loop 6
       lb_algo wlc
       lb_kind dr
       persistence_timeout 600
       protocol TCP
       real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
       real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
   }

   Backup load balancer node

   vrrp_instance RGW {
       state BACKUP # might not be necessary?
       priority 99
       advert_int 1
       interface eno1
       virtual_router_id 50
       unicast_src_ip 10.8.128.53 80
       unicast_peer {
              10.8.128.43
              }
       authentication {
           auth_type PASS
           auth_pass 1111
       }
       virtual_ipaddress {
           192.168.1.20
       }
       track_script {
         chk_haproxy
       }
   }
   virtual_server 192.168.1.20 80 eno1 { #populate correct interface
       delay_loop 6
       lb_algo wlc
       lb_kind dr
       persistence_timeout 600
       protocol TCP
       real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
       real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
   }

3. Enable and start the keepalived service:

   [root@haproxy]# systemctl enable keepalived
   [root@haproxy]# systemctl start keepalived

1. Install haproxy.

   [root@haproxy]# yum install haproxy

2. Configure haproxy for SELinux and HTTP.

   [root@haproxy]# vim /etc/firewalld/services/haproxy-http.xml

   Add the following lines:

   <?xml version="1.0" encoding="utf-8"?>
   <service>
   <short>HAProxy-HTTP</short>
   <description>HAProxy load-balancer</description>
   <port protocol="tcp" port="80"/>
   </service>

   As root, assign the correct SELinux context and file permissions to the haproxy-http.xml file.

   [root@haproxy]# cd /etc/firewalld/services
   [root@haproxy]# restorecon haproxy-http.xml
   [root@haproxy]# chmod 640 haproxy-http.xml

3. If you intend to use HTTPS, configure haproxy for SELinux and HTTPS.

   [root@haproxy]# vim /etc/firewalld/services/haproxy-https.xml

   Add the following lines:

   <?xml version="1.0" encoding="utf-8"?>
   <service>
   <short>HAProxy-HTTPS</short>
   <description>HAProxy load-balancer</description>
   <port protocol="tcp" port="443"/>
   </service>

   As root, assign the correct SELinux context and file permissions to the haproxy-https.xml file.

   # cd /etc/firewalld/services
   # restorecon haproxy-https.xml
   # chmod 640 haproxy-https.xml

4. If you intend to use HTTPS, generate keys for SSL. If you do not have a certificate, you may use a self-signed certificate. To generate a key, see to Generating a New Key and Certificate section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

   Finally, put the certificate and key into a PEM file.

   [root@haproxy]# cat example.com.crt example.com.key > example.com.pem
   [root@haproxy]# cp example.com.pem /etc/ssl/private/

5. Configure haproxy.

   [root@haproxy]# vim /etc/haproxy/haproxy.cfg

   The global and defaults may remain unchanged. After the defaults section, you will need to configure frontend and backend sections. For example:

   frontend http_web
       bind *:80
       mode http
       default_backend rgw
   
   frontend rgw­-https
     bind *:443 ssl crt /etc/ssl/private/example.com.pem
     default_backend rgw
   
   backend rgw
       balance roundrobin
       mode http
       server  rgw1 10.0.0.71:80 check
       server  rgw2 10.0.0.80:80 check

   For a detailed discussion of HAProxy configuration, refer to HAProxy Configuration.

6. Enable/start haproxy

   [root@haproxy]# systemctl enable haproxy
   [root@haproxy]# systemctl start haproxy

1. S3 static web hosting uses Ceph Object Gateway instances that are separate and distinct from instances used for standard S3/Swift API use cases.
2. Gateway instances hosting S3 static web sites should have separate, non-overlapping domain names from the standard S3/Swift API gateway instances.
3. Gateway instances hosting S3 static web sites should use separate public-facing IP addresses from the standard S3/Swift API gateway instances.
4. Gateway instances hosting S3 static web sites load balance, and if necessary terminate SSL, using HAProxy/keepalived.

1. Create an S3 bucket. The bucket name MAY be the same as the website’s domain name. For example, mysite.com may have a bucket name of mysite.com. This is required for AWS, but it is NOT required for Ceph. See DNS Settings for details.
2. Upload the static website content to the bucket. Contents may include HTML, CSS, client-side JavaScript, images, audio/video content and other downloadable files. A website MUST have an index.html file and MAY have error.html file.
3. Verify the website’s contents. At this point, only the creator of the bucket will have access to the contents.
4. Set permissions on the files so that they are publicly readable.

Procedure

1. Calculate the recommended number of shards. To do so, use the following formula:

   number of objects expected in a bucket / 100,000

   Note that maximum number of shards is 65521.

2. Add rgw_override_bucket_index_max_shards to the Ceph configuration file:

   rgw_override_bucket_index_max_shards = value

   Replace value with the recommended number of shards calculated in the previous step, for example:

   rgw_override_bucket_index_max_shards = 12

   • To configure bucket index sharding for all instances of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the [global] section.
   • To configure bucket index sharding only for a particular instance of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the instance.

3. Restart the Ceph Object Gateway:

   # systemctl restart ceph-radosgw.target

Procedure

1. Calculate the recommended number of shards. To do so, use the following formula:

   number of objects expected in a bucket / 100,000

   Note that maximum number of shards is 65521.

2. Extract the zone group configuration to the zonegroup.json file:

   $ radosgw-admin zonegroup get > zonegroup.json

3. In the zonegroup.json file, set the bucket_index_max_shards setting for each named zone.

   bucket_index_max_shards = VALUE

   Replace value with the recommended number of shards calculated in the previous step, for example:

   bucket_index_max_shards = 12

4. Reset the zone group:

   $ radosgw-admin zonegroup set < zonegroup.json

5. Update the period:

   $ radosgw-admin period update --commit

Procedure

1. Back the original bucket index up:

   radosgw-admin bi list --bucket=bucket > bucket.list.backup

   Replace:

   • bucket with the name of the bucket to reshard

   For example, for a bucket named data, enter:

   $ radosgw-admin bi list --bucket=data > data.list.backup

2. Reshard the bucket index:

   radosgw-admin bucket reshard --bucket=bucket
   --num-shards=number

   Replace:

   • bucket with the name of the bucket to reshard
   • number with the new number of shards

   For example, for a bucket named data and the required number of shards being 100, enter:

   $ radosgw-admin bucket reshard --bucket=data
   --num-shards=100

3. If you use Red Hat Ceph Storage 3.1 and previous versions, remove stale bucket entries as described in the Cleaning stale instances after resharding section.

Procedure

1. List stale instances:

   $ radosgw-admin reshard stale-instances list

2. Clean the stale instances:

   $ radosgw-admin reshard stale-instances rm

Procedure

1. Rename a user:

   radosgw-admin user rename --uid=current-user-name --new-uid=new-user-name

   For example, to rename user1 to user2:

   # radosgw-admin user rename --uid=user1 --new-uid=user2
   
   {
       "user_id": "user2",
       "display_name": "user 2",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "user2",
               "access_key": "59EKHI6AI9F8WOW8JQZJ",
               "secret_key": "XH0uY3rKCUcuL73X0ftjXbZqUbk0cavD11rD8MsA"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

   If a user is inside a tenant, specify both the user name and the tenant:

   Syntax

   radosgw-admin user rename --uid user-name --new-uid new-user-name --tenant tenant

   For example, to rename user1 to user2 inside a test tenant:

   Example

   # radosgw-admin user rename --uid=test$user1 --new-uid=test$user2 --tenant test
   
   1000 objects processed in tvtester1. Next marker 80_tVtester1_99
   2000 objects processed in tvtester1. Next marker 64_tVtester1_44
   3000 objects processed in tvtester1. Next marker 48_tVtester1_28
   4000 objects processed in tvtester1. Next marker 2_tVtester1_74
   5000 objects processed in tvtester1. Next marker 14_tVtester1_53
   6000 objects processed in tvtester1. Next marker 87_tVtester1_61
   7000 objects processed in tvtester1. Next marker 6_tVtester1_57
   8000 objects processed in tvtester1. Next marker 52_tVtester1_91
   9000 objects processed in tvtester1. Next marker 34_tVtester1_74
   9900 objects processed in tvtester1. Next marker 9_tVtester1_95
   1000 objects processed in tvtester2. Next marker 82_tVtester2_93
   2000 objects processed in tvtester2. Next marker 64_tVtester2_9
   3000 objects processed in tvtester2. Next marker 48_tVtester2_22
   4000 objects processed in tvtester2. Next marker 32_tVtester2_42
   5000 objects processed in tvtester2. Next marker 16_tVtester2_36
   6000 objects processed in tvtester2. Next marker 89_tVtester2_46
   7000 objects processed in tvtester2. Next marker 70_tVtester2_78
   8000 objects processed in tvtester2. Next marker 51_tVtester2_41
   9000 objects processed in tvtester2. Next marker 33_tVtester2_32
   9900 objects processed in tvtester2. Next marker 9_tVtester2_83
   {
       "user_id": "test$user2",
       "display_name": "User 2",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "test$user2",
               "access_key": "user2",
               "secret_key": "123456789"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Verify that the user has been renamed successfully:

   Syntax

   radosgw-admin user info --uid=new-user-name

   For example:

   Example

   # radosgw-admin user info --uid=user2

   If a user is inside a tenant, use the tenant$user-name format:

   radosgw-admin user info --uid=tenant$new-user-name

   # radosgw-admin user info --uid=test$user2

1. Find the access key for the specific user:

   [root@master-zone]# radosgw-admin user info --uid=<testid>

   The access key is the "access_key" value in the output, for example:

   $ radosgw-admin user info --uid=johndoe
   {
       "user_id": "johndoe",
       ...
       "keys": [
           {
               "user": "johndoe",
               "access_key": "0555b35654ad1656d804",
               "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
           }
       ],
       ...
   }

2. Specify the user ID and the access key from the previous step to remove the access key:

   [root@master-zone]# radosgw-admin key rm --uid=<user_id> --access-key <access_key>

   For example:

   [root@master-zone]# radosgw-admin key rm --uid=johndoe --access-key 0555b35654ad1656d804

Procedure

1. List the buckets:

   radosgw-admin bucket list

   For example, note a bucket from the output:

   # radosgw-admin bucket list
   [
       "34150b2e9174475db8e191c188e920f6/swcontainer",
       "s3bucket1",
       "34150b2e9174475db8e191c188e920f6/swimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
       "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
       "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
       "34150b2e9174475db8e191c188e920f6/postimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
       "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
   ]

2. Rename the bucket:

   radosgw-admin bucket link --bucket=original-name --bucket-new-name=new-name --uid=user-ID

   For example, to rename the s3bucket1 bucket to s3newb:

   # radosgw-admin bucket link --bucket=s3bucket1 --bucket-new-name=s3newb --uid=testuser

   If the bucket is inside a tenant, specify the tenant as well:

   radosgw-admin bucket link --bucket=tenant/original-name --bucket-new-name=new-name --uid=tenant$user-ID

   For example:

   # radosgw-admin bucket link --bucket=test/s3bucket1 --bucket-new-name=s3newb --uid=test$testuser

3. Verify the bucket was renamed:

   radosgw-admin bucket list

   For example, a bucket named s3newb exists now:

   # radosgw-admin bucket list
   [
       "34150b2e9174475db8e191c188e920f6/swcontainer",
       "34150b2e9174475db8e191c188e920f6/swimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
       "s3newb",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
       "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
       "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
       "34150b2e9174475db8e191c188e920f6/postimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
       "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
   ]

Procedure

1. To generate a list of objects that hold data within a bucket:

   Syntax

   radosgw-admin bucket radoslist --bucket BUCKET_NAME

   Example

   [root@rgw ~]# radosgw-admin bucket radoslist --bucket mybucket

   Note

   If the BUCKET_NAME is omitted, then all objects in all buckets are displayed.

2. To generate a list of orphans for a pool:

   [root@rgw ~]# rgw-orphan-list

   Example

   Available pools:
       .rgw.root
       default.rgw.control
       default.rgw.meta
       default.rgw.log
       default.rgw.buckets.index
       default.rgw.buckets.data
       rbd
       default.rgw.buckets.non-ec
       ma.rgw.control
       ma.rgw.meta
       ma.rgw.log
       ma.rgw.buckets.index
       ma.rgw.buckets.data
       ma.rgw.buckets.non-ec
   Which pool do you want to search for orphans?

   Enter the pool name to search for orphans.

   Important

   A data pool must be specified when using the rgw-orphan-list command, and not a metadata pool.

3. Review the orphan objects in the list.

4. To remove orphan objects:

   Syntax

   rados -p POOL_NAME rm OBJECT_NAME

   Example

   [root@rgw ~]# rados -p default.rgw.buckets.data rm myobject

   Warning

   Verify you are removing the correct objects. Executing the rados rm command will remove data from the storage cluster.

Procedure

1. Check the bucket index of a specific bucket:

   Syntax

   radosgw-admin bucket check --bucket=BUCKET_NAME

   Example

   [root@rgw ~]# radosgw-admin bucket check --bucket=mybucket

2. Fix the inconsistencies in the bucket index, including removal of orphaned objects:

   Syntax

   radosgw-admin bucket check --fix --bucket=BUCKET_NAME

   Example

   [root@rgw ~]# radosgw-admin bucket check --fix --bucket=mybucket

Procedure

1. Create a s3 bucket.
2. Create a SNS topic for http,amqp or kafka protocol.

3. Create a s3 bucket notification for s3:objectCreate and s3:objectRemove events:

   Example

   client.put_bucket_notification_configuration(
      Bucket=bucket_name,
      NotificationConfiguration={
          'TopicConfigurations': [
              {
                  'Id': notification_name,
                  'TopicArn': topic_arn,
                  'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*']
              }]})

4. Create s3 objects in the bucket.
5. Verify the object creation events at the http or rabbitmq or kafka receiver.
6. Delete the objects.
7. Verify the object deletion events at the http or rabbitmq or kafka receiver.

Procedure

1. Create a JSON file for lifecycle configuration:

   Example

   [user@client ~]$ vi lifecycle.json

2. Add the specific lifecycle configuration rules in the file:

   Example

   {
   	"Rules": [
           {
   		    "Filter": {
   			    "Prefix": "images/"
   		    },
   		    "Status": "Enabled",
   		    "Expiration": {
   			    "Days": 1
   		    },
   		    "ID": "ImageExpiration"
   	    }
       ]
   }

   The lifecycle configuration example expires objects in the images directory that are more than 1 day old.

3. Set the lifecycle configuration on the bucket:

   Syntax

   aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

   Example

   [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

   In this example, the lifecycle.json file exists in the current directory.

Procedure

1. Create a JSON file for the lifecycle configuration:

   Example

   [user@client ~]$ vi lifecycle.json

2. Add the specific lifecycle configuration rules to the file:

   Example

   {
   	"Rules": [
           {
   		    "Filter": {
   			    "Prefix": "images/"
   		    },
   		    "Status": "Enabled",
   		    "Expiration": {
   			    "Days": 1
   		    },
   		    "ID": "ImageExpiration"
   	    },
   		{
   			"Filter": {
   				"Prefix": "docs/"
   			},
   			"Status": "Enabled",
   			"Expiration": {
   				"Days": 30
   			},
   			"ID": "DocsExpiration"
   		}
   	]
   }

3. Update the lifecycle configuration on the bucket:

   Syntax

   aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

   Example

   [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

Procedure

1. List bucket lifecycle progress:

   Example

   [root@rgw ~]# radosgw-admin lc list
   
   [
      {
            “bucket”: “:testbucket:8b63d584-9ea1-4cf3-8443-a6a15beca943.54187.1”,
            “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
            “status” : “UNINITIAL”
      },
      {
            “bucket”: “:testbucket1:8b635499-9e41-4cf3-8443-a6a15345943.54187.2”,
            “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
            “status” : “UNINITIAL”
      }
   ]

   The bucket lifecycle processing status can be one of the following:

   • UNINITIAL - The process has not run yet.
   • PROCESSING - The process is currently running.
   • COMPLETE - The process has completed.

2. Optional: You can manually process bucket lifecycle policies:

   1. Process the lifecycle policy for a single bucket:

      Syntax

      radosgw-admin lc process --bucket=BUCKET_NAME

      Example

      [root@rgw ~]# radosgw-admin lc process --bucket=testbucket1

   2. Process all bucket lifecycle policies immediately:

      Example

      [root@rgw ~]# radosgw-admin lc process

Procedure

1. Set the lifecycle expiration time:

   Syntax

   ceph config set client.rgw rgw_lifecycle_work_time %D:%D-%D:%D

   Replace %d:%d-%d:%d with start_hour:start_minute-end_hour:end_minute.

   Example

   [root@rgw ~]# ceph config set client.rgw rgw_lifecycle_work_time 06:00-08:00

Procedure

1. Create a new data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Example

   [root@rgw ~]# ceph osd pool create test.hot.data

2. Add a new storage class:

   Syntax

   radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

   Example

   [root@rgw ~]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class hot.test
   {
           "key": "default-placement",
           "val": {
               "name": "default-placement",
               "tags": [],
               "storage_classes": [
                   "STANDARD",
                   "hot.test"
               ]
           }
       }

3. Provide the zone placement information for the new storage class:

   Syntax

   radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

   Example

   [root@rgw ~]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class hot.test --data-pool test.hot.data
   {
              "key": "default-placement",
              "val": {
                  "index_pool": "test_zone.rgw.buckets.index",
                  "storage_classes": {
                      "STANDARD": {
                          "data_pool": "test.hot.data"
                      },
                      "hot.test": {
                          "data_pool": "test.hot.data",
                     }
                  },
                  "data_extra_pool": "",
                  "index_type": 0
              }

   Note

   Consider setting the compression_type when creating cold or archival data storage pools with write once.

4. Enable rgw application on the data pool:

   Syntax

   ceph osd pool application enable POOL_NAME rgw

   Example

   [root@rgw ~] ceph osd pool application enable test.hot.data rgw
   enabled application 'rgw' on pool 'test.hot.data'

5. Restart all the rgw daemons.

6. Create a bucket:

   Example

   [root@rgw ~]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

7. Add the object:

   Example

   [root@rgw ~]# aws --endpoint=http://1x.7x.2xx.1xx:80 s3api put-object --bucket testbucket10  --key compliance-upload --body /root/test2.txt

8. Create a second data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Example

   [root@rgw ~]# ceph osd pool create test.cold.data

9. Add a new storage class:

   Syntax

   radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

   Example

   [root@rgw ~]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class cold.test
   {
           "key": "default-placement",
           "val": {
               "name": "default-placement",
               "tags": [],
               "storage_classes": [
                   "STANDARD",
                   "cold.test"
               ]
           }
       }

10. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [root@rgw ~]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class cold.test --data-pool test.cold.data

11. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [root@rgw ~] ceph osd pool application enable test.cold.data rgw
    enabled application 'rgw' on pool 'test.cold.data'

12. Restart all the rgw daemons.

13. To view zone group configuration, execute:

    Syntax

    radosgw-admin zonegroup get
    {
        "id": "3019de59-ddde-4c5c-b532-7cdd29de09a1",
        "name": "default",
        "api_name": "default",
        "is_master": "true",
        "endpoints": [],
        "hostnames": [],
        "hostnames_s3website": [],
        "master_zone": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "zones": [
            {
                "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
                "name": "default",
                "endpoints": [],
                "log_meta": "false",
                "log_data": "false",
                "bucket_index_max_shards": 11,
                "read_only": "false",
                "tier_type": "",
                "sync_from_all": "true",
                "sync_from": [],
                "redirect_zone": ""
            }
        ],
        "placement_targets": [
            {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "hot.test",
                    "cold.test",
                    "STANDARD"
                ]
            }
        ],
        "default_placement": "default-placement",
        "realm_id": "",
        "sync_policy": {
            "groups": []
        }
    }

14. To view the zone configuration,execute:

    Syntax

    radosgw-admin zone get
    {
        "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "name": "default",
        "domain_root": "default.rgw.meta:root",
        "control_pool": "default.rgw.control",
        "gc_pool": "default.rgw.log:gc",
        "lc_pool": "default.rgw.log:lc",
        "log_pool": "default.rgw.log",
        "intent_log_pool": "default.rgw.log:intent",
        "usage_log_pool": "default.rgw.log:usage",
        "roles_pool": "default.rgw.meta:roles",
        "reshard_pool": "default.rgw.log:reshard",
        "user_keys_pool": "default.rgw.meta:users.keys",
        "user_email_pool": "default.rgw.meta:users.email",
        "user_swift_pool": "default.rgw.meta:users.swift",
        "user_uid_pool": "default.rgw.meta:users.uid",
        "otp_pool": "default.rgw.otp",
        "system_key": {
            "access_key": "",
            "secret_key": ""
        },
        "placement_pools": [
            {
                "key": "default-placement",
                "val": {
                    "index_pool": "default.rgw.buckets.index",
                    "storage_classes": {
                        "cold.test": {
                            "data_pool": "test.cold.data"
                        },
                        "hot.test": {
                            "data_pool": "test.hot.data"
                        },
                        "STANDARD": {
                            "data_pool": "default.rgw.buckets.data"
                        }
                    },
                    "data_extra_pool": "default.rgw.buckets.non-ec",
                    "index_type": 0
                }
            }
        ],
        "realm_id": "",
        "notif_pool": "default.rgw.log:notif"
    }

15. Create a bucket:

    Example

    [root@rgw ~]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

16. Create a JSON file for lifecycle configuration:

    Example

    [root@rgw ~]# vi lifecycle.json

17. Add the specific lifecyle configuration rule in the file:

    Example

    {
        "Rules": [
            {
                "Filter": {
                    "Prefix": ""
                },
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    },
     {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    }
                ],
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration"
            }
        ]
    }

    The lifecycle configuration example shows an object that will transition from the default STANDARD storage class to the hot.test storage class after 5 days, again transitions after 20 days to the cold.test storage class, and finally expires after 365 days in the cold.test storage class.

18. Set the lifecycle configuration on the bucket:

    Example

    [root@rgw ~] aws s3api put-bucket-lifecycle-configuration --bucket testbucket20 --lifecycle-configuration file://lifecycle.json

19. Retrieve the lifecycle configuration on the bucket:

    Example

    [root@rgw ~]aws s3api get-bucket-lifecycle-configuration --bucket testbucket20
    {
        "Rules": [
            {
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration",
                "Prefix": "",
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    },
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    }
                ]
            }
        ]
    }

Procedure

1. To view the queue of objects awaiting garbage collection:

   Example

   [root@rgw ~] radosgw-admin gc list

Procedure

1. Open /etc/ceph/ceph.conf for editing.

2. Set the value of rgw_gc_max_concurrent_io to 20, and the value of rgw_gc_max_trim_chunk to 64.

   rgw_gc_max_concurrent_io = 20
   rgw_gc_max_trim_chunk = 64

3. Restart the Ceph Object Gateway to allow the changed settings to take effect.
4. Monitor the storage cluster during GC activity to verify that the increased values do not adversely affect performance.

Procedure

1. Open /etc/ceph/ceph.conf for editing.

2. To increase the number of threads to run in parallel, set the value of rgw_lc_max_worker to a value between 3 and 9:

   Syntax

   rgw_lc_max_worker = VALUE

   Example

   rgw_lc_max_worker = 7

3. To increase the number of threads in each thread’s work pool, set the value of rgw_lc_max_wp_worker to a value between 3 and 9:

   Syntax

   rgw_lc_max_wp_worker = VALUE

   Example

   rgw_lc_max_wp_worker = 7

4. Restart the Ceph Object Gateway to allow the changed settings to take effect.
5. Monitor the storage cluster to verify that the increased values do not adversely affect performance.

Procedure

1. Generate a 30 character seed from the urandom Linux device file and store it in the shell variable SEED:

   Example

   [user@host ~]$ SEED=$(head -10 /dev/urandom | sha512sum | cut -b 1-30)

2. Print the seed by running echo on the SEED variable:

   Example

   [user@host ~]$ echo $SEED
   492dedb20cf51d1405ef6a1316017e

   Configure the one-time password generator and the back-end MFA system to use the same seed.

Procedure

1. Create a new MFA TOTP token:

   Syntax

   radosgw-admin mfa create --uid=USERID --totp-serial=SERIAL --totp-seed=SEED --totp-seed-type=SEED_TYPE --totp-seconds=TOTP_SECONDS --totp-window=TOTP_WINDOW

   Set USERID to the user name to set up MFA on, set SERIAL to a string that represents the ID for the TOTP token, and set SEED to a hexadecimal or base32 value that is used to calculate the TOTP. The following settings are optional: Set the SEED_TYPE to hex or base32, set TOTP_SECONDS to the timeout in seconds, or set TOTP_WINDOW to the number of TOTP tokens to check before and after the current token when validating tokens.

   Example

   [root@mon ~]# radosgw-admin mfa create --uid=johndoe --totp-serial=MFAtest --totp-seed=492dedb20cf51d1405ef6a1316017e

Procedure

1. Test the TOTP token PIN to verify that TOTP functions correctly:

   Syntax

   radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

   Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the latest PIN from the one-time password generator.

   Example

   [root@mon ~] # radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
   ok

   If this is the first time you have tested the PIN, it may fail. If it fails, resynchronize the token. See Resynchronizing a multi-factor authentication token in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

Procedure

1. Resynchronize a multi-factor authentication TOTP token in case of time skew or failed checks.

   This requires passing in two consecutive pins: the previous pin, and the current pin.

   Syntax

   radosgw-admin mfa resync --uid=USERID --totp-serial=SERIAL --totp-pin=PREVIOUS_PIN --totp=pin=CURRENT_PIN

   Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, set PREVIOUS_PIN to the user’s previous PIN, and set CURRENT_PIN to the user’s current PIN.

   Example

   radosgw-admin mfa resync --uid=johndoe --totp-serial=MFAtest --totp-pin=802017 --totp-pin=439996

2. Verify the token was successfully resynchronized by testing a new PIN:

   Syntax

   radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

   Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the user’s PIN.

   Example

   [root@mon ~]# radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
   ok

Procedure

1. List MFA TOTP tokens:

   Syntax

   radosgw-admin mfa list --uid=USERID

   Set USERID to the user name MFA is set up on.

   Example

   [root@mon ~]# radosgw-admin mfa list --uid=johndoe
   {
       "entries": [
           {
               "type": 2,
               "id": "MFAtest",
               "seed": "492dedb20cf51d1405ef6a1316017e",
               "seed_type": "hex",
               "time_ofs": 0,
               "step_size": 30,
               "window": 2
           }
       ]
   }

Procedure

1. Show the MFA TOTP token:

   Syntax

   radosgw-admin mfa get --uid=USERID --totp-serial=SERIAL

   Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

Procedure

1. Delete an MFA TOTP token:

   Syntax

   radosgw-admin mfa remove --uid=USERID --totp-serial=SERIAL

   Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

   Example

   [root@mon ~]# radosgw-admin mfa remove --uid=johndoe --totp-serial=MFAtest

2. Verify the MFA TOTP token was deleted:

   Syntax

   radosgw-admin mfa get --uid=_USERID_ --totp-serial=_SERIAL_

   Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

   Example

   [root@mon ~]# radosgw-admin mfa get --uid=johndoe --totp-serial=MFAtest
   MFA serial id not found

1. Remove the zonegroup and the zone:

   Example

   [root@master-zone]# radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
   [root@master-zone]# radosgw-admin zone delete --rgw-zone=default
   [root@master-zone]# radosgw-admin zonegroup delete --rgw-zonegroup=default

2. Update and commit the period if the cluster is in a multi-site configuration:

   Example

   [root@master-zone]# radosgw-admin period update --commit

3. Delete the default pools in your Ceph storage cluster if they exist.

   Example

   [root@master-zone]# ceph osd pool delete default.rgw.control default.rgw.control --yes-i-really-really-mean-it
   [root@master-zone]# ceph osd pool delete default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
   [root@master-zone]# ceph osd pool delete default.rgw.log default.rgw.log --yes-i-really-really-mean-it
   [root@master-zone]# ceph osd pool delete default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it

   Important

   After deleting the pools, restart the Ceph Object Gateway process.