Search

Chapter 3. Administration

download PDF

Administrators can manage the Ceph Object Gateway using the radosgw-admin command-line interface.

3.1. Administrative Data Storage

A Ceph Object Gateway stores administrative data in a series of pools defined in an instance’s zone configuration. For example, the buckets, users, user quotas and usage statistics discussed in the subsequent sections are stored in pools in the Ceph Storage Cluster. By default, Ceph Object Gateway will create the following pools and map them to the default zone.

  • .rgw.root
  • .default.rgw.control
  • .default.rgw.meta
  • .default.rgw.log
  • .default.rgw.buckets.index
  • .default.rgw.buckets.data
  • .default.rgw.buckets.non-ec

You should consider creating these pools manually so that you can set the CRUSH ruleset and the number of placement groups. In a typical configuration, the pools that store the Ceph Object Gateway’s administrative data will often use the same CRUSH ruleset and use fewer placement groups, because there are 10 pools for the administrative data. See Pools and the Storage Strategies guide for Red Hat Ceph Storage 4 for additional details.

Also see Ceph Placement Groups (PGs) per Pool Calculator for placement group calculation details. The mon_pg_warn_max_per_osd setting warns you if assign too many placement groups to a pool (i.e., 300 by default). You may adjust the value to suit your needs and the capabilities of your hardware where n is the maximum number of PGs per OSD.

mon_pg_warn_max_per_osd = n

3.2. Creating Storage Policies

The Ceph Object Gateway stores the client bucket and object data by identifying placement targets, and storing buckets and objects in the pools associated with a placement target. If you don’t configure placement targets and map them to pools in the instance’s zone configuration, the Ceph Object Gateway will use default targets and pools, for example, default_placement.

Storage policies give Ceph Object Gateway clients a way of accessing a storage strategy, that is, the ability to target a particular type of storage, for example, SSDs, SAS drives, SATA drives. A particular way of ensuring durability, replication, erasure coding, and so on. For details, see the Storage Strategies guide for Red Hat Ceph Storage 4.

To create a storage policy, use the following procedure:

  1. Create a new pool .rgw.buckets.special with the desired storage strategy. For example, a pool customized with erasure-coding, a particular CRUSH ruleset, the number of replicas, and the pg_num and pgp_num count.
  2. Get the zone group configuration and store it in a file, for example, zonegroup.json:

    Syntax

    [root@master-zone]# radosgw-admin zonegroup --rgw-zonegroup=<zonegroup_name> get > zonegroup.json

    Example

    [root@master-zone]# radosgw-admin zonegroup --rgw-zonegroup=default get > zonegroup.json

  3. Add a special-placement entry under placement_target in the zonegroup.json file.

    {
    	"name": "default",
    	"api_name": "",
    	"is_master": "true",
    	"endpoints": [],
    	"hostnames": [],
    	"master_zone": "",
    	"zones": [{
    		"name": "default",
    		"endpoints": [],
    		"log_meta": "false",
    		"log_data": "false",
    		"bucket_index_max_shards": 11
    	}],
    	"placement_targets": [{
    		"name": "default-placement",
    		"tags": []
    	}, {
    		"name": "special-placement",
    		"tags": []
    	}],
    	"default_placement": "default-placement"
    }
  4. Set the zone group with the modified zonegroup.json file:

    [root@master-zone]# radosgw-admin zonegroup set < zonegroup.json
  5. Get the zone configuration and store it in a file, for example, zone.json:

    [root@master-zone]# radosgw-admin zone get > zone.json
  6. Edit the zone file and add the new placement policy key under placement_pool:

    {
    	"domain_root": ".rgw",
    	"control_pool": ".rgw.control",
    	"gc_pool": ".rgw.gc",
    	"log_pool": ".log",
    	"intent_log_pool": ".intent-log",
    	"usage_log_pool": ".usage",
    	"user_keys_pool": ".users",
    	"user_email_pool": ".users.email",
    	"user_swift_pool": ".users.swift",
    	"user_uid_pool": ".users.uid",
    	"system_key": {
    		"access_key": "",
    		"secret_key": ""
    	},
    	"placement_pools": [{
    		"key": "default-placement",
    		"val": {
    			"index_pool": ".rgw.buckets.index",
    			"data_pool": ".rgw.buckets",
    			"data_extra_pool": ".rgw.buckets.extra"
    		}
    	}, {
    		"key": "special-placement",
    		"val": {
    			"index_pool": ".rgw.buckets.index",
    			"data_pool": ".rgw.buckets.special",
    			"data_extra_pool": ".rgw.buckets.extra"
    		}
    	}]
    }
  7. Set the new zone configuration.

    [root@master-zone]# radosgw-admin zone set < zone.json
  8. Update the zone group map.

    [root@master-zone]# radosgw-admin period update --commit

    The special-placement entry is listed as a placement_target.

To specify the storage policy when making a request:

Example:

$ curl -i http://10.0.0.1/swift/v1/TestContainer/file.txt -X PUT -H "X-Storage-Policy: special-placement" -H "X-Auth-Token: AUTH_rgwtxxxxxx"

3.3. Creating indexless buckets

You can configure a placement target where created buckets do not use the bucket index to store objects index; that is, indexless buckets. Placement targets that do not use data replication or listing might implement indexless buckets. Indexless buckets provides a mechanism in which the placement target does not track objects in specific buckets. This removes a resource contention that happens whenever an object write happens and reduces the number of round trips that Ceph Object Gateway needs to make to the Ceph storage cluster. This can have a positive effect on concurrent operations and small object write performance.

Warning

It has been observed that the Ceph Object Gateway daemon crashes when performing operations on a bucket having indexless placement policy. Hence, Red Hat does not recommend having this placement policy.

Important

The bucket index will not reflect the correct state of the bucket, and listing these buckets will not correctly return their list of objects. This affects multiple features. Specifically, these buckets will not be synced in a multi-zone environment because the bucket index is not used to store change information. Red Hat recommends not to use S3 object versioning on indexless buckets, because the bucket index is necessary for this feature.

Note

Using indexless buckets removes the limit of the max number of objects in a single bucket.

Note

Objects in indexless buckets cannot be listed from NFS.

Prerequisites

  • A running and healthy Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Root-level access to a Ceph Object Gateway node.

Procedure

  1. Add a new placement target to the zonegroup:

    Example

    [root@rgw ~]# radosgw-admin zonegroup placement add --rgw-zonegroup="default" \
      --placement-id="indexless-placement"

  2. Add a new placement target to the zone:

    Example

    [root@rgw ~]# radosgw-admin zone placement add --rgw-zone="default" \
       --placement-id="indexless-placement" \
       --data-pool="default.rgw.buckets.data" \
       --index-pool="default.rgw.buckets.index" \
       --data_extra_pool="default.rgw.buckets.non-ec" \
       --placement-index-type="indexless"

  3. Set the zonegroup’s default placement to indexless-placement:

    Example

    [root@rgw ~]# radosgw-admin zonegroup placement default --placement-id "indexless-placement"

    In this example, the buckets created in the indexless-placement target will be indexless buckets.

  4. Update and commit the period if the cluster is in a multi-site configuration:

    Example

    [root@rgw ~]# radosgw-admin period update --commit

  5. Restart the Ceph Object Gateway daemon for the change to take effect:

    Example

    [root@rgw ~]# systemctl restart ceph-radosgw.target

3.4. Configuring Bucket sharding

The Ceph Object Gateway stores bucket index data in the index pool (index_pool), which defaults to .rgw.buckets.index. When the client puts many objects—​hundreds of thousands to millions of objects—​in a single bucket without having set quotas for the maximum number of objects per bucket, the index pool can suffer significant performance degradation.

Bucket index sharding helps prevent performance bottlenecks when allowing a high number of objects per bucket. Starting with Red Hat Ceph Storage 4.1, default number of bucket index shards, bucket_index_max_shards, has been changed from 1 to 11. This change increases the amount of write throughput for small buckets, and delays the onset of dynamic resharding. This change affects only the new buckets and deployments.

Red Hat recommends to have the shard count as the nearest prime number to the calculated shard count. The bucket index shards that are prime numbers tend to work better in evenly distributing bucket index entries across the shards. For example, 7001 bucket index shards is better than 7000 since the former is prime.

To configure bucket index sharding:

To reshard a bucket:

3.4.1. Bucket sharding limitations

Important

Use the following limitations with caution. There are implications related to your hardware selections, so you should always discuss these requirements with your Red Hat account team.

Maximum number of objects in one bucket before it needs sharding

Red Hat recommends a maximum of 102,400 objects per bucket index shard. To take full advantage of sharding, provide a sufficient number of OSDs in the Ceph Object Gateway bucket index pool to get maximum parallelism.

Note

Ceph OSDs currently warn when any key range in indexed storage exceeds 200,000. As a consequence, if you approach the number of 200,000 objects per shard, you will get such warnings. In some setups, the value might be larger, and is adjustable.

Maximum number of objects when using sharding

The default number of bucket index shards for dynamic bucket resharding is 1999. You can change this value up to 65521 shards. A value of 1999 bucket index shards gives 204697600 total objects in the bucket, a value of 65521 shards gives 6709350400 objects.

Note

Based on prior testing, the maximum number of bucket index shards currently supported is 65521. Red Hat quality assurance has NOT performed full scalability testing on bucket sharding.

Important

If the number of bucket index shards exceeds 1999, ordinary S3 clients might not be able to list bucket contents. The custom clients can ask for unordered listing, which scales to any number of shards.

3.4.2. Bucket lifecycle parallel thread processing

A new feature in Red Hat Ceph Storage 4.1 allows for parallel thread processing of bucket lifecycles. This parallelization scales with the number of Ceph Object Gateway instance, and replaces the in-order index shard enumeration with a number sequence. The default locking timeout has been extended from 60 seconds to 90 seconds. New tunable options have been added to tune lifecycle worker threads to run in parallel for each Ceph Object Gateway instance.

rgw_lc_max_worker

This option specifies the number of lifecycle worker thread to run in parallel, thereby processing bucket and index shards simultaneously. The default value for the rgw_lc_max_worker option is 3.

rgw_lc_max_wp_worker

This option specifies the number of threads in each lifecycle worker’s work pool. This option can help accelerate processing each bucket. The default value for the rgw_lc_max_wp_worker option is 3.

Additional Resources

3.4.3. Configuring Bucket Index Sharding in Simple Configurations

To enable and configure bucket index sharding on all new buckets, use the rgw_override_bucket_index_max_shards parameter. Set the parameter to:

  • 0 to disable bucket index sharding. This is the default value.
  • A value greater than 0 to enable bucket sharding and to set the maximum number of shards.

Prerequisites

Procedure

  1. Calculate the recommended number of shards. To do so, use the following formula:

    number of objects expected in a bucket / 100,000

    Note that maximum number of shards is 65521.

  2. Add rgw_override_bucket_index_max_shards to the Ceph configuration file:

    rgw_override_bucket_index_max_shards = value

    Replace value with the recommended number of shards calculated in the previous step, for example:

    rgw_override_bucket_index_max_shards = 12
    • To configure bucket index sharding for all instances of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the [global] section.
    • To configure bucket index sharding only for a particular instance of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the instance.
  3. Restart the Ceph Object Gateway:

    # systemctl restart ceph-radosgw.target

3.4.4. Configuring Bucket Index sharding in Multisite Configurations

In multisite configurations, each zone can have a different index_pool setting to manage failover. To configure a consistent shard count for zones in one zone group, set the bucket_index_max_shards setting in the configuration for that zone group. Set the parameter to:

Set the parameter to:

  • 0 to disable bucket index sharding, the default value of bucket_index_max_shards is 11.
  • A value greater than 0 to enable bucket sharding and to set the maximum number of shards.
Note

Mapping the index pool (for each zone, if applicable) to a CRUSH ruleset of SSD-based OSDs might also help with bucket index performance.

Prerequisites

Procedure

  1. Calculate the recommended number of shards. To do so, use the following formula:

    number of objects expected in a bucket / 100,000

    Note that maximum number of shards is 65521.

  2. Extract the zone group configuration to the zonegroup.json file:

    $ radosgw-admin zonegroup get > zonegroup.json
  3. In the zonegroup.json file, set the bucket_index_max_shards setting for each named zone.

    bucket_index_max_shards = VALUE

    Replace value with the recommended number of shards calculated in the previous step, for example:

    bucket_index_max_shards = 12
  4. Reset the zone group:

    $ radosgw-admin zonegroup set < zonegroup.json
  5. Update the period:

    $ radosgw-admin period update --commit

3.4.5. Dynamic Bucket Index Resharding

The process for dynamic bucket resharding periodically checks all the Ceph Object Gateway buckets and detects buckets that require resharding. If a bucket has grown larger than the value specified in the rgw_max_objs_per_shard parameter, the Ceph Object Gateway reshards the bucket dynamically in the background. The default value for rgw_max_objs_per_shard is 100k objects per shard.

Note

The default value is based on experience with bucket indexes stored on spinning disk. In more modern setups with bucket indexes on flash media, the value for maximum objects per bucket index shard might be higher.

Note

Dynamic bucket index resharding works as expected on the upgraded single-site configuration without any modification to the zone or the zone group. A single site-configuration can be any of the following:

  • A default zone configuration with no realm.
  • A non-default configuration with at least one realm.
  • A multi-realm single-site configuration.

Prerequisites

Procedure

  • To enable dynamic bucket index resharding:

    1. Set the rgw_dynamic_resharding setting in the Ceph configuration file to true, which is the default value.
    2. Optional. Change the following parameters in the Ceph configuration file if needed:

      • rgw_reshard_num_logs: The number of shards for the resharding log. The default value is 16.
      • rgw_reshard_bucket_lock_duration: The duration of the lock on a bucket during resharding. The default value is 360 seconds.
      • rgw_dynamic_resharding: Enables or disables dynamic resharding. The default value is true.
      • rgw_max_objs_per_shard: The maximum number of objects per shard. The default value is 100000 objects per shard.
      • rgw_reshard_thread_interval: The maximum time between rounds of reshard thread processing. The default value is 600 seconds.
  • To add a bucket to the resharding queue:

    radosgw-admin reshard add --bucket bucket --num-shards number

    Replace:

    • bucket with the name of the bucket to reshard
    • number with the new number of shards

    For example:

    $ radosgw-admin reshard add --bucket data --num-shards 10
  • To list the resharding queue:

    $ radosgw-admin reshard list
  • To check bucket resharding status:

    radosgw-admin reshard status --bucket bucket

    Replace:

    • bucket with the name of the bucket to reshard

    For example:

    $ radosgw-admin reshard status --bucket data
  • To process entries on the resharding queue immediately:

    $ radosgw-admin reshard process
  • To cancel pending bucket resharding:

    radosgw-admin reshard cancel --bucket bucket

    Replace:

    • bucket with the name of the pending bucket

    For example:

    $ radosgw-admin reshard cancel --bucket data
    Important

    You can only cancel pending resharding operations. Do not cancel ongoing resharding operations.

  • If you use Red Hat Ceph Storage 3.1 and previous versions, remove stale bucket entries as described in the Cleaning stale instances after resharding section.

3.4.6. Manual Bucket Index Resharding

If a bucket has grown larger than the initial configuration was optimized for, reshard the bucket index pool by using the radosgw-admin bucket reshard command. This command:

  • Creates a new set of bucket index objects for the specified bucket.
  • Distributes object entries across these bucket index objects.
  • Creates a new bucket instance.
  • Links the new bucket instance with the bucket so that all new index operations go through the new bucket indexes.
  • Prints the old and the new bucket ID to the command output.
Important

Use this procedure only in simple configurations. To reshard buckets in multi-site configurations, see Manually Resharding Buckets with Multi-site.

Prerequisites

Procedure

  1. Back the original bucket index up:

    radosgw-admin bi list --bucket=bucket > bucket.list.backup

    Replace:

    • bucket with the name of the bucket to reshard

    For example, for a bucket named data, enter:

    $ radosgw-admin bi list --bucket=data > data.list.backup
  2. Reshard the bucket index:

    radosgw-admin bucket reshard --bucket=bucket
    --num-shards=number

    Replace:

    • bucket with the name of the bucket to reshard
    • number with the new number of shards

    For example, for a bucket named data and the required number of shards being 100, enter:

    $ radosgw-admin bucket reshard --bucket=data
    --num-shards=100
  3. If you use Red Hat Ceph Storage 3.1 and previous versions, remove stale bucket entries as described in the Cleaning stale instances after resharding section.

3.4.7. Cleaning stale instances after resharding

In Red Hat Ceph Storage 3.1 and previous versions, the resharding process does not clean stale instances of bucket entries automatically. These stale instances can impact performance of the cluster if they are not cleaned manually.

Important

Use this procedure only in simple configurations not in multi-site clusters.

Prerequisites

  • Ceph Object Gateway installed.

Procedure

  1. List stale instances:

    $ radosgw-admin reshard stale-instances list
  2. Clean the stale instances:

    $ radosgw-admin reshard stale-instances rm

3.5. Enabling Compression

The Ceph Object Gateway supports server-side compression of uploaded objects using any of Ceph’s compression plugins. These include:

  • zlib: Supported.
  • snappy: Technology Preview.
  • zstd: Technology Preview.
Note

The snappy and zstd compression plugins are Technology Preview features and as such they are not fully supported, as Red Hat has not completed quality assurance testing on them yet.

Configuration

To enable compression on a zone’s placement target, provide the --compression=<type> option to the radosgw-admin zone placement modify command. The compression type refers to the name of the compression plugin to use when writing new object data.

Each compressed object stores the compression type. Changing the setting does not hinder the ability to decompress existing compressed objects, nor does it force the Ceph Object Gateway to recompress existing objects.

This compression setting applies to all new objects uploaded to buckets using this placement target.

To disable compression on a zone’s placement target, provide the --compression=<type> option to the radosgw-admin zone placement modify command and specify an empty string or none.

For example:

$ radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib
{
...
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": "zlib"
            }
        }
    ],
...
}

After enabling or disabling compression, restart the Ceph Object Gateway instance so the change will take effect.

Note

Ceph Object Gateway creates a default zone and a set of pools. For production deployments, see the Ceph Object Gateway for Production guide, more specifically, the Creating a Realm section first. See also Multisite.

Statistics

While all existing commands and APIs continue to report object and bucket sizes based on their uncompressed data, the radosgw-admin bucket stats command includes compression statistics for a given bucket.

$ radosgw-admin bucket stats --bucket=<name>
{
...
    "usage": {
        "rgw.main": {
            "size": 1075028,
            "size_actual": 1331200,
            "size_utilized": 592035,
            "size_kb": 1050,
            "size_kb_actual": 1300,
            "size_kb_utilized": 579,
            "num_objects": 104
        }
    },
...
}

The size_utilized and size_kb_utilized fields represent the total size of compressed data in bytes and kilobytes respectively.

3.6. User Management

Ceph Object Storage user management refers to users that are client applications of the Ceph Object Storage service; not the Ceph Object Gateway as a client application of the Ceph Storage Cluster. You must create a user, access key and secret to enable client applications to interact with the Ceph Object Gateway service.

There are two user types:

  • User: The term 'user' reflects a user of the S3 interface.
  • Subuser: The term 'subuser' reflects a user of the Swift interface. A subuser is associated to a user .

You can create, modify, view, suspend and remove users and subusers.

Important

When managing users in a multi-site deployment, ALWAYS execute the radosgw-admin command on a Ceph Object Gateway node within the master zone of the master zone group to ensure that users synchronize throughout the multi-site cluster. DO NOT create, modify or delete users on a multi-site cluster from a secondary zone or a secondary zone group. This document uses [root@master-zone]# as a command line convention for a host in the master zone of the master zone group.

In addition to creating user and subuser IDs, you may add a display name and an email address for a user. You can specify a key and secret, or generate a key and secret automatically. When generating or specifying keys, note that user IDs correspond to an S3 key type and subuser IDs correspond to a swift key type. Swift keys also have access levels of read, write, readwrite and full.

User management command-line syntax generally follows the pattern user <command> <user-id> where <user-id> is either the --uid= option followed by the user’s ID (S3) or the --subuser= option followed by the user name (Swift). For example:

[root@master-zone]# radosgw-admin user <create|modify|info|rm|suspend|enable|check|stats> <--uid={id}|--subuser={name}> [other-options]

Additional options may be required depending on the command you execute.

3.6.1. Multi Tenancy

In Red Hat Ceph Storage 2 and later, the Ceph Object Gateway supports multi-tenancy for both the S3 and Swift APIs, where each user and bucket lies under a "tenant." Multi tenancy prevents namespace clashing when multiple tenants are using common bucket names, such as "test", "main" and so forth.

Each user and bucket lies under a tenant. For backward compatibility, a "legacy" tenant with an empty name is added. Whenever referring to a bucket without specifically specifying a tenant, the Swift API will assume the "legacy" tenant. Existing users are also stored under the legacy tenant, so they will access buckets and objects the same way as earlier releases.

Tenants as such do not have any operations on them. They appear and and disappear as needed, when users are administered. In order to create, modify, and remove users with explicit tenants, either an additional option --tenant is supplied, or a syntax "<tenant>$<user>" is used in the parameters of the radosgw-admin command.

To create a user testx$tester for S3, execute the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester \
                    --display-name "Test User" --access_key TESTER \
                    --secret test123 user create

To create a user testx$tester for Swift, execute one of the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester \
                    --display-name "Test User" --subuser tester:swift \
                    --key-type swift --access full subuser create

[root@master-zone]# radosgw-admin key create --subuser 'testx$tester:swift' \
                    --key-type swift --secret test123
Note

The subuser with explicit tenant had to be quoted in the shell.

3.6.2. Create a User

Use the user create command to create an S3-interface user. You MUST specify a user ID and a display name. You may also specify an email address. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

[root@master-zone]# radosgw-admin user create --uid=<id> \
[--key-type=<type>] [--gen-access-key|--access-key=<key>]\
[--gen-secret | --secret=<key>] \
[--email=<email>] --display-name=<name>

For example:

[root@master-zone]# radosgw-admin user create --uid=janedoe --display-name="Jane Doe" --email=jane@example.com
{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}
Important

Check the key output. Sometimes radosgw-admin generates a JSON escape (\) character, and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character (\), encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually.

3.6.3. Create a Subuser

To create a subuser (Swift interface), you must specify the user ID (--uid={username}), a subuser ID and the access level for the subuser. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

Note

full is not readwrite, as it also includes the access control policy.

[root@master-zone]# radosgw-admin subuser create --uid={uid} --subuser={uid} --access=[ read | write | readwrite | full ]

For example:

[root@master-zone]# radosgw-admin subuser create --uid=janedoe --subuser=janedoe:swift --access=full
{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "janedoe:swift",
          "permissions": "full-control"}],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

3.6.4. Get User Information

To get information about a user, specify user info and the user ID (--uid={username}).

[root@master-zone]# radosgw-admin user info --uid=janedoe

To get information about a tenanted user, specify both the user ID and the name of the tenant.

[root@master-zone]# radosgw-admin user info --uid=janedoe --tenant=test

3.6.5. Modify User Information

To modify information about a user, you must specify the user ID (--uid={username}) and the attributes you want to modify. Typical modifications are to keys and secrets, email addresses, display names and access levels. For example:

[root@master-zone]# radosgw-admin user modify --uid=janedoe --display-name="Jane E. Doe"

To modify subuser values, specify subuser modify and the subuser ID. For example:

[root@master-zone]# radosgw-admin subuser modify --subuser=janedoe:swift --access=full

3.6.6. Enable and Suspend Users

When you create a user, the user is enabled by default. However, you may suspend user privileges and re-enable them at a later time. To suspend a user, specify user suspend and the user ID.

[root@master-zone]# radosgw-admin user suspend --uid=johndoe

To re-enable a suspended user, specify user enable and the user ID. :

[root@master-zone]# radosgw-admin user enable --uid=johndoe
Note

Disabling the user disables the subuser.

3.6.7. Remove a User

When you remove a user, the user and subuser are removed from the system. However, you may remove just the subuser if you wish. To remove a user (and subuser), specify user rm and the user ID.

[root@master-zone]# radosgw-admin user rm --uid=<uid> [--purge-keys] [--purge-data]

For example:

[root@master-zone]# radosgw-admin user rm --uid=johndoe --purge-data

To remove the subuser only, specify subuser rm and the subuser name.

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:swift --purge-keys

Options include:

  • Purge Data: The --purge-data option purges all data associated to the UID.
  • Purge Keys: The --purge-keys option purges all keys associated to the UID.

3.6.8. Remove a Subuser

When you remove a sub user, you are removing access to the Swift interface. The user will remain in the system. The Ceph Object Gateway To remove the subuser, specify subuser rm and the subuser ID.

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:test

Options include:

  • Purge Keys: The --purge-keys option purges all keys associated to the UID.

3.6.9. Rename a User

To change a name of a user, use the radosgw-admin user rename command. The time that this command takes depends on the number of buckets and objects that the user has. If the number is large, Red Hat recommends to use the command in the Screen utility provided by the screen package.

Prerequisites

  • A working Ceph cluster
  • root or sudo access
  • Installed Ceph Object Gateway

Procedure

  1. Rename a user:

    radosgw-admin user rename --uid=current-user-name --new-uid=new-user-name

    For example, to rename user1 to user2:

    # radosgw-admin user rename --uid=user1 --new-uid=user2
    
    {
        "user_id": "user2",
        "display_name": "user 2",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "user2",
                "access_key": "59EKHI6AI9F8WOW8JQZJ",
                "secret_key": "XH0uY3rKCUcuL73X0ftjXbZqUbk0cavD11rD8MsA"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

    If a user is inside a tenant, specify both the user name and the tenant:

    Syntax

    radosgw-admin user rename --uid user-name --new-uid new-user-name --tenant tenant

    For example, to rename user1 to user2 inside a test tenant:

    Example

    # radosgw-admin user rename --uid=test$user1 --new-uid=test$user2 --tenant test
    
    1000 objects processed in tvtester1. Next marker 80_tVtester1_99
    2000 objects processed in tvtester1. Next marker 64_tVtester1_44
    3000 objects processed in tvtester1. Next marker 48_tVtester1_28
    4000 objects processed in tvtester1. Next marker 2_tVtester1_74
    5000 objects processed in tvtester1. Next marker 14_tVtester1_53
    6000 objects processed in tvtester1. Next marker 87_tVtester1_61
    7000 objects processed in tvtester1. Next marker 6_tVtester1_57
    8000 objects processed in tvtester1. Next marker 52_tVtester1_91
    9000 objects processed in tvtester1. Next marker 34_tVtester1_74
    9900 objects processed in tvtester1. Next marker 9_tVtester1_95
    1000 objects processed in tvtester2. Next marker 82_tVtester2_93
    2000 objects processed in tvtester2. Next marker 64_tVtester2_9
    3000 objects processed in tvtester2. Next marker 48_tVtester2_22
    4000 objects processed in tvtester2. Next marker 32_tVtester2_42
    5000 objects processed in tvtester2. Next marker 16_tVtester2_36
    6000 objects processed in tvtester2. Next marker 89_tVtester2_46
    7000 objects processed in tvtester2. Next marker 70_tVtester2_78
    8000 objects processed in tvtester2. Next marker 51_tVtester2_41
    9000 objects processed in tvtester2. Next marker 33_tVtester2_32
    9900 objects processed in tvtester2. Next marker 9_tVtester2_83
    {
        "user_id": "test$user2",
        "display_name": "User 2",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "test$user2",
                "access_key": "user2",
                "secret_key": "123456789"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

  2. Verify that the user has been renamed successfully:

    Syntax

    radosgw-admin user info --uid=new-user-name

    For example:

    Example

    # radosgw-admin user info --uid=user2

    If a user is inside a tenant, use the tenant$user-name format:

    radosgw-admin user info --uid=tenant$new-user-name
    # radosgw-admin user info --uid=test$user2

Additional Resources

  • The screen(1) manual page

3.6.10. Create a Key

To create a key for a user, you must specify key create. For a user, specify the user ID and the s3 key type. To create a key for subuser, you must specify the subuser ID and the swift keytype. For example:

[root@master-zone]# radosgw-admin key create --subuser=johndoe:swift --key-type=swift --gen-secret
{ "user_id": "johndoe",
  "rados_uid": 0,
  "display_name": "John Doe",
  "email": "john@example.com",
  "suspended": 0,
  "subusers": [
     { "id": "johndoe:swift",
       "permissions": "full-control"}],
  "keys": [
    { "user": "johndoe",
      "access_key": "QFAMEDSJP5DEKJO0DDXY",
      "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
  "swift_keys": [
    { "user": "johndoe:swift",
      "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}

3.6.11. Add and Remove Access Keys

Users and subusers must have access keys to use the S3 and Swift interfaces. When you create a user or subuser and you do not specify an access key and secret, the key and secret get generated automatically. You may create a key and either specify or generate the access key and/or secret. You may also remove an access key and secret. Options include:

  • --secret=<key> specifies a secret key (e.g,. manually generated).
  • --gen-access-key generates random access key (for S3 user by default).
  • --gen-secret generates a random secret key.
  • --key-type=<type> specifies a key type. The options are: swift, s3

To add a key, specify the user:

[root@master-zone]# radosgw-admin key create --uid=johndoe --key-type=s3 --gen-access-key --gen-secret

You may also specify a key and a secret.

To remove an access key, you need to specify the user and the key:

  1. Find the access key for the specific user:

    [root@master-zone]# radosgw-admin user info --uid=<testid>

    The access key is the "access_key" value in the output, for example:

    $ radosgw-admin user info --uid=johndoe
    {
        "user_id": "johndoe",
        ...
        "keys": [
            {
                "user": "johndoe",
                "access_key": "0555b35654ad1656d804",
                "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
            }
        ],
        ...
    }
  2. Specify the user ID and the access key from the previous step to remove the access key:

    [root@master-zone]# radosgw-admin key rm --uid=<user_id> --access-key <access_key>

    For example:

    [root@master-zone]# radosgw-admin key rm --uid=johndoe --access-key 0555b35654ad1656d804

3.6.12. Add and Remove Admin Capabilities

The Ceph Storage Cluster provides an administrative API that enables users to execute administrative functions via the REST API. By default, users DO NOT have access to this API. To enable a user to exercise administrative functionality, provide the user with administrative capabilities.

To add administrative capabilities to a user, execute the following:

[root@master-zone]# radosgw-admin caps add --uid={uid} --caps={caps}

You can add read, write or all capabilities to users, buckets, metadata and usage (utilization). For example:

--caps="[users|buckets|metadata|usage|zone]=[*|read|write|read, write]"

For example:

[root@master-zone]# radosgw-admin caps add --uid=johndoe --caps="users=*"

To remove administrative capabilities from a user, execute the following:

[root@master-zone]# radosgw-admin caps remove --uid=johndoe --caps={caps}

3.7. Quota Management

The Ceph Object Gateway enables you to set quotas on users and buckets owned by users. Quotas include the maximum number of objects in a bucket and the maximum storage size in megabytes.

  • Bucket: The --bucket option allows you to specify a quota for buckets the user owns.
  • Maximum Objects: The --max-objects setting allows you to specify the maximum number of objects. A negative value disables this setting.
  • Maximum Size: The --max-size option allows you to specify a quota for the maximum number of bytes. A negative value disables this setting.
  • Quota Scope: The --quota-scope option sets the scope for the quota. The options are bucket and user. Bucket quotas apply to buckets a user owns. User quotas apply to a user.
Important

Buckets with a large number of objects can cause serious performance issues. The recommended maximum number of objects in a one bucket is 100,000. To increase this number, configure bucket index sharding. See Section 3.4, “Configuring Bucket sharding” for details.

3.7.1. Set User Quotas

Before you enable a quota, you must first set the quota parameters. For example:

[root@master-zone]# radosgw-admin quota set --quota-scope=user --uid=<uid> [--max-objects=<num objects>] [--max-size=<max size>]

For example:

radosgw-admin quota set --quota-scope=user --uid=johndoe --max-objects=1024 --max-size=1024

A negative value for num objects and / or max size means that the specific quota attribute check is disabled.

3.7.2. Enable and Disable User Quotas

Once you set a user quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=user --uid=<uid>

You may disable an enabled user quota. For example:

[root@master-zone]# radosgw-admin quota disable --quota-scope=user --uid=<uid>

3.7.3. Set bucket quotas

Bucket quotas apply to the buckets owned by the specified uid. They are independent of the user.

Syntax

radosgw-admin quota set --uid=USER_ID --quota-scope=bucket --bucket=BUCKET_NAME [--max-objects=NUMBER_OF_OBJECTS] [--max-size=MAXIMUM_SIZE_IN_BYTES]

A negative value for NUMBER_OF_OBJECTS, MAXIMUM_SIZE_IN_BYTES, or both means that the specific quota attribute check is disabled.

3.7.4. Enable and Disable Bucket Quotas

Once you set a bucket quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=bucket --uid=<uid>

You may disable an enabled bucket quota. For example:

[root@master-zone]# radosgw-admin quota disable --quota-scope=bucket --uid=<uid>

3.7.5. Get Quota Settings

You may access each user’s quota settings via the user information API. To read user quota setting information with the CLI interface, execute the following:

# radosgw-admin user info --uid=<uid>

To get quota settings for a tenanted user, specify the user ID and the name of the tenant:

+ radosgw-admin user info --uid=_user-id_ --tenant=_tenant_

3.7.6. Update Quota Stats

Quota stats get updated asynchronously. You can update quota statistics for all users and all buckets manually to retrieve the latest quota stats.

[root@master-zone]# radosgw-admin user stats --uid=<uid> --sync-stats

3.7.7. Get User Quota Usage Stats

To see how much of the quota a user has consumed, execute the following:

# radosgw-admin user stats --uid=<uid>
Note

You should execute radosgw-admin user stats with the --sync-stats option to receive the latest data.

3.7.8. Quota Cache

Quota statistics are cached for each Ceph Gateway instance. If there are multiple instances, then the cache can keep quotas from being perfectly enforced, as each instance will have a different view of the quotas. The options that control this are rgw bucket quota ttl, rgw user quota bucket sync interval and rgw user quota sync interval. The higher these values are, the more efficient quota operations are, but the more out-of-sync multiple instances will be. The lower these values are, the closer to perfect enforcement multiple instances will achieve. If all three are 0, then quota caching is effectively disabled, and multiple instances will have perfect quota enforcement. See Chapter 4, Configuration Reference for more details on these options.

3.7.9. Reading and Writing Global Quotas

You can read and write quota settings in a zonegroup map. To get a zonegroup map:

[root@master-zone]# radosgw-admin global quota get

The global quota settings can be manipulated with the global quota counterparts of the quota set, quota enable, and quota disable commands, for example:

[root@master-zone]# radosgw-admin global quota set --quota-scope bucket --max-objects 1024
[root@master-zone]# radosgw-admin global quota enable --quota-scope bucket
Note

In a multi-site configuration, where there is a realm and period present, changes to the global quotas must be committed using period update --commit. If there is no period present, the Ceph Object Gateways must be restarted for the changes to take effect.

3.8. Usage

The Ceph Object Gateway logs usage for each user. You can track user usage within date ranges too.

Options include:

  • Start Date: The --start-date option allows you to filter usage stats from a particular start date (format: yyyy-mm-dd[HH:MM:SS]).
  • End Date: The --end-date option allows you to filter usage up to a particular date (format: yyyy-mm-dd[HH:MM:SS]).
  • Log Entries: The --show-log-entries option allows you to specify whether or not to include log entries with the usage stats (options: true | false).
Note

You can specify time with minutes and seconds, but it is stored with 1 hour resolution.

3.8.1. Show Usage

To show usage statistics, specify the usage show. To show usage for a particular user, you must specify a user ID. You may also specify a start date, end date, and whether or not to show log entries.

# radosgw-admin usage show \
                --uid=johndoe --start-date=2012-03-01 \
                --end-date=2012-04-01

You may also show a summary of usage information for all users by omitting a user ID.

# radosgw-admin usage show --show-log-entries=false

3.8.2. Trim Usage

With heavy use, usage logs can begin to take up storage space. You can trim usage logs for all users and for specific users. You may also specify date ranges for trim operations.

[root@master-zone]# radosgw-admin usage trim --start-date=2010-01-01 \
                    --end-date=2010-12-31

[root@master-zone]# radosgw-admin usage trim --uid=johndoe
[root@master-zone]# radosgw-admin usage trim --uid=johndoe --end-date=2013-12-31

3.9. Bucket management

As a storage administrator, when using the Ceph Object Gateway you can manage buckets by moving them between users and renaming them. Also, you can find orphan or leaky objects within the Ceph Object Gateway that can occur over the lifetime of a storage cluster.

3.9.1. Moving buckets

The radosgw-admin bucket utility provides the ability to move buckets between users. To do so, link the bucket to a new user and change the ownership of the bucket to the new user.

You can move buckets:

3.9.1.1. Prerequisites

  • A running Red Hat Ceph Storage cluster
  • Ceph Object Gateway is installed
  • A bucket
  • Various tenanted and non-tenanted users

3.9.1.2. Moving buckets between non-tenanted users

The radosgw-admin bucket chown command provides the ability to change the ownership of buckets and all objects they contain from one user to another. To do so, unlink a bucket from the current user, link it to a new user, and change the ownership of the bucket to the new user.

Procedure

  1. Link the bucket to a new user:

    radosgw-admin bucket link --uid=user --bucket=bucket

    Replace:

    • user with the user name of the user to link the bucket to
    • bucket with the name of the bucket

    For example, to link the data bucket to the user named user2:

    # radosgw-admin bucket link --uid=user2 --bucket=data
  2. Verify that the bucket has been linked to user2 successfully:

    # radosgw-admin bucket list --uid=user2
    [
        "data"
    ]
  3. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --uid=user --bucket=bucket

    Replace:

    • user with the user name of the user to change the bucket ownership to
    • bucket with the name of the bucket

    For example, to change the ownership of the data bucket to user2:

    # radosgw-admin bucket chown --uid=user2 --bucket=data
  4. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=data

3.9.1.3. Moving buckets between tenanted users

You can move buckets between one tenanted user to another.

Procedure

  1. Link the bucket to a new user:

    radosgw-admin bucket link --bucket=current-tenant/bucket --uid=new-tenant$user

    Replace:

    • current-tenant with the name of the tenant the bucket is
    • bucket with the name of the bucket to link
    • new-tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to link the data bucket from the test tenant to the user named user2 in the test2 tenant:

    # radosgw-admin bucket link --bucket=test/data --uid=test2$user2
  2. Verify that the bucket has been linked to user2 successfully:

    # radosgw-admin bucket list --uid=test$user2
    [
        "data"
    ]
  3. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --bucket=new-tenant/bucket --uid=new-tenant$user

    Replace:

    • bucket with the name of the bucket to link
    • new-tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to change the ownership of the data bucket to the user2 inside the test2 tenant:

    # radosgw-admin bucket chown --bucket='test2/data' --uid='test$tuser2'
  4. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=test2/data

3.9.1.4. Moving buckets from non-tenanted users to tenanted users

You can move buckets from a non-tenanted user to a tenanted user.

Procedure

  1. Optional. If you do not already have multiple tenants, you can create them by enabling rgw_keystone_implicit_tenants and accessing the Ceph Object Gateway from an external tenant:

    Open and edit the Ceph configuration file, by default /etc/ceph/ceph.conf. Enable the rgw_keystone_implicit_tenants option:

    rgw_keystone_implicit_tenants = true

    Access the Ceph Object Gateway from an eternal tenant using either the s3cmd or swift command:

    # swift list

    Or use s3cmd:

    # s3cmd ls

    The first access from an external tenant creates an equivalent Ceph Object Gateway user.

  2. Move a bucket to a tenanted user:

    radosgw-admin bucket link --bucket=/bucket --uid='tenant$user'

    Replace:

    • bucket with the name of the bucket
    • tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to move the data bucket to the tenanted-user inside the test tenant:

    # radosgw-admin bucket link --bucket=/data --uid='test$tenanted-user'
  3. Verify that the data bucket has been linked to tenanted-user successfully:

    # radosgw-admin bucket list --uid='test$tenanted-user'
    [
        "data"
    ]
  4. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --bucket='tenant/bucket name' --uid='tenant$user'

    Replace:

    • bucket with the name of the bucket
    • tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to change the ownership of the data bucket to tenanted-user that is inside the test tenant:

    # radosgw-admin bucket chown --bucket='test/data' --uid='test$tenanted-user'
  5. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=test/data

3.9.2. Renaming buckets

You can rename buckets.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • A bucket.

Procedure

  1. List the buckets:

    radosgw-admin bucket list

    For example, note a bucket from the output:

    # radosgw-admin bucket list
    [
        "34150b2e9174475db8e191c188e920f6/swcontainer",
        "s3bucket1",
        "34150b2e9174475db8e191c188e920f6/swimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
        "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
        "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
        "34150b2e9174475db8e191c188e920f6/postimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
        "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
    ]
  2. Rename the bucket:

    radosgw-admin bucket link --bucket=original-name --bucket-new-name=new-name --uid=user-ID

    For example, to rename the s3bucket1 bucket to s3newb:

    # radosgw-admin bucket link --bucket=s3bucket1 --bucket-new-name=s3newb --uid=testuser

    If the bucket is inside a tenant, specify the tenant as well:

    radosgw-admin bucket link --bucket=tenant/original-name --bucket-new-name=new-name --uid=tenant$user-ID

    For example:

    # radosgw-admin bucket link --bucket=test/s3bucket1 --bucket-new-name=s3newb --uid=test$testuser
  3. Verify the bucket was renamed:

    radosgw-admin bucket list

    For example, a bucket named s3newb exists now:

    # radosgw-admin bucket list
    [
        "34150b2e9174475db8e191c188e920f6/swcontainer",
        "34150b2e9174475db8e191c188e920f6/swimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
        "s3newb",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
        "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
        "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
        "34150b2e9174475db8e191c188e920f6/postimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
        "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
    ]

3.9.3. Finding orphan and leaky objects

A healthy storage cluster does not have any orphan or leaky objects, but in some cases orphan or leaky objects can occur. For example, if the Ceph Object Gateway goes down in the middle of an operation, this can cause some objects to become orphans. Also, an undiscovered bug can cause orphan objects to occur.

Starting with Red Hat Ceph Storage 4.1, storage administrators can see how the Ceph Object Gateway objects map to the RADOS objects. The radosgw-admin command provides you a new tool to search for and produce a list of these potential orphan or leaky objects. Using the radoslist subcommand will display objects stored within buckets, or all buckets in the storage cluster. The rgw-orphan-list script will display orphan objects within a pool.

Warning

The rgw-orphan-list command is still experimental. Cautiously and carefully evaluate the objects listed by it before removing any using the rados rm command.

Important

The radoslist subcommand is replacing the deprecated orphans find and orphans finish subcommands.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A running Ceph Object Gateway.

Procedure

  1. To generate a list of objects that hold data within a bucket:

    Syntax

    radosgw-admin bucket radoslist --bucket BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin bucket radoslist --bucket mybucket

    Note

    If the BUCKET_NAME is omitted, then all objects in all buckets are displayed.

  2. To generate a list of orphans for a pool:

    [root@rgw ~]# rgw-orphan-list

    Example

    Available pools:
        .rgw.root
        default.rgw.control
        default.rgw.meta
        default.rgw.log
        default.rgw.buckets.index
        default.rgw.buckets.data
        rbd
        default.rgw.buckets.non-ec
        ma.rgw.control
        ma.rgw.meta
        ma.rgw.log
        ma.rgw.buckets.index
        ma.rgw.buckets.data
        ma.rgw.buckets.non-ec
    Which pool do you want to search for orphans?

    Enter the pool name to search for orphans.

    Important

    A data pool must be specified when using the rgw-orphan-list command, and not a metadata pool.

  3. Review the orphan objects in the list.
  4. To remove orphan objects:

    Syntax

    rados -p POOL_NAME rm OBJECT_NAME

    Example

    [root@rgw ~]# rados -p default.rgw.buckets.data rm myobject

    Warning

    Verify you are removing the correct objects. Executing the rados rm command will remove data from the storage cluster.

Additional Resources

  • See the Finding Orphan Objects section in the Red Hat Ceph Storage 3 Object Gateway Administration Guide for more details on the legacy radosgw-admin orphans find subcommand.

3.9.4. Managing bucket index entries

You can manage the bucket index entries of the Ceph Object Gateway in a Red Hat Ceph Storage cluster using the radosgw-admin bucket check subcommand.

Each bucket index entry related to a piece of a multipart upload object is matched against its corresponding .meta index entry. There should be one .meta entry for all the pieces of a given multipart upload. If it fails to find a corresponding .meta entry for a piece, it lists out the "orphaned" piece entries in a section of the output.

The stats for the bucket are stored in the bucket index headers. This phase loads those headers and also iterates through all the plain object entries in the bucket index and recalculates the stats. It then displays the actual and calculated stats in sections labeled "existing_header" and "calculated_header" respectively, so they can be compared.

If you use the --fix option with the bucket check subcommand, it removes the "orphaned" entries from the bucket index and also overwrites the existing stats in the header with those that it calculated. It causes all entries, including the multiple entries used in versioning, to be listed in a section of the output.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A running Ceph Object Gateway.
  • A bucket created.

Procedure

  1. Check the bucket index of a specific bucket:

    Syntax

    radosgw-admin bucket check --bucket=BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin bucket check --bucket=mybucket

  2. Fix the inconsistencies in the bucket index, including removal of orphaned objects:

    Syntax

    radosgw-admin bucket check --fix --bucket=BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin bucket check --fix --bucket=mybucket

3.9.5. Bucket notifications

Bucket notifications provide a way to send information out of the Ceph Object Gateway when certain events happen in the bucket. Bucket notifications can be sent to HTTP, AMQP0.9.1 and Kafka endpoints.

A notification entry must be created to send bucket notifications for events on a specific bucket and to a specific topic. A bucket notification can be created on a subset of event types or by default for all event types. The bucket notification can filter out events based on key prefix or suffix, regular expression matching the keys, and on the metadata attributes attached to the object, or the object tags. Bucket notifications have a REST API to provide configuration and control interfaces for the bucket notification mechanism.

Note

The bucket notifications API are enabled by default. If rgw_enable_apis configuration parameter is explicitly set, ensure that s3 and pubsub are included. To verify this, run ceph config get mon.* rgw_enable_apis command.

Additional Resources

3.9.6. Creating bucket notifications

Create bucket notifications at the bucket level. The notification configuration has the Red Hat Ceph Storage Object Gateway S3 events, ObjectCreated and ObjectRemoved. These need to be published and the destination to send the bucket notifications. Bucket notifications are S3 operations.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A running HTTP server, RabbitMQ server, or a Kafka server.
  • Root-level access.
  • Installation of the Red Hat Ceph Storage Object Gateway.
  • User access key and secret key.
  • Endpoint parameters.
Important

Red Hat supports ObjectCreate events, such as, put, post, multipartUpload, and copy. Red Hat also supports ObjectRemove events, such as, object_delete and s3_multi_object_delete.

Procedure

  1. Create a s3 bucket.
  2. Create a SNS topic for http,amqp or kafka protocol.
  3. Create a s3 bucket notification for s3:objectCreate and s3:objectRemove events:

    Example

    client.put_bucket_notification_configuration(
       Bucket=bucket_name,
       NotificationConfiguration={
           'TopicConfigurations': [
               {
                   'Id': notification_name,
                   'TopicArn': topic_arn,
                   'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*']
               }]})

  4. Create s3 objects in the bucket.
  5. Verify the object creation events at the http or rabbitmq or kafka receiver.
  6. Delete the objects.
  7. Verify the object deletion events at the http or rabbitmq or kafka receiver.

3.9.7. Additional Resources

3.10. Bucket lifecycle

As a storage administrator, you can use a bucket lifecycle configuration to manage your objects so they are stored effectively throughout their lifetime. For example you can transition objects to less expensive storage classes, archive, or even delete them based on your use case.

Note

The radosgw-admin lc reshard command is deprecated in Red Hat Ceph Storage 3.3 and not supported in Red Hat Ceph Storage 4 and later releases.

3.10.1. Creating a lifecycle management policy

You can manage bucket lifecycle policy configuration using standard S3 operations rather than using the radosgw-admin command. RADOS Gateway supports only a subset of the Amazon S3 API policy language applied to buckets. The lifecycle configuration contains one or more rules defined for a set of bucket objects.

Prerequisites

  • A running Red Hat Storage cluster.
  • Installation of the Ceph Object Gateway.
  • Root-level access to a Ceph Object Gateway node.
  • An S3 bucket created.
  • An S3 user created with user access.
  • Access to a Ceph Object Gateway client with the AWS CLI package installed.

Procedure

  1. Create a JSON file for lifecycle configuration:

    Example

    [user@client ~]$ vi lifecycle.json

  2. Add the specific lifecycle configuration rules in the file:

    Example

    {
    	"Rules": [
            {
    		    "Filter": {
    			    "Prefix": "images/"
    		    },
    		    "Status": "Enabled",
    		    "Expiration": {
    			    "Days": 1
    		    },
    		    "ID": "ImageExpiration"
    	    }
        ]
    }

    The lifecycle configuration example expires objects in the images directory that are more than 1 day old.

  3. Set the lifecycle configuration on the bucket:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

    In this example, the lifecycle.json file exists in the current directory.

Verification

  • Retrieve the lifecycle configuration for the bucket:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api get-bucket-lifecycle-configuration --bucket BUCKET_NAME

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api get-bucket-lifecycle-configuration --bucket testbucket
    {
    	"Rules": [
            {
    		    "Expiration": {
    			    "Days": 1
    		    },
    		    "ID": "ImageExpiration",
    		    "Filter": {
    			    "Prefix": "images/"
    		    },
    		    "Status": "Enabled"
    	    }
        ]
    }

  • Optional: From the Ceph Object Gateway node, log into the Cephadm shell and retrieve the bucket lifecycle configuration:

    Syntax

    radosgw-admin lc get --bucket=BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin lc get --bucket=testbucket
    {
    	"prefix_map": {
    		"images/": {
    			"status": true,
    			"dm_expiration": false,
    			"expiration": 1,
    			"noncur_expiration": 0,
    			"mp_expiration": 0,
    			"transitions": {},
    			"noncur_transitions": {}
    		}
    	},
    	"rule_map": [
            {
    		"id": "ImageExpiration",
    		"rule": {
    			"id": "ImageExpiration",
    			"prefix": "",
    			"status": "Enabled",
    			"expiration": {
    				"days": "1",
    				"date": ""
    			},
    			"mp_expiration": {
    				"days": "",
    				"date": ""
    			},
    			"filter": {
    				"prefix": "images/",
    				"obj_tags": {
    					"tagset": {}
    				}
    			},
    			"transitions": {},
    			"noncur_transitions": {},
    			"dm_expiration": false
    		}
    	}
      ]
    }

Additional Resources

3.10.2. Deleting a lifecycle management policy

You can delete the lifecycle management policy for a specified bucket by using the s3api delete-bucket-lifecycle command.

Prerequisites

  • A running Red Hat Storage cluster.
  • Installation of the Ceph Object Gateway.
  • Root-level access to a Ceph Object Gateway node.
  • An S3 bucket created.
  • An S3 user created with user access.
  • Access to a Ceph Object Gateway client with the AWS CLI package installed.

Procedure

  • Delete a lifecycle configuration:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api delete-bucket-lifecycle --bucket BUCKET_NAME

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api delete-bucket-lifecycle --bucket testbucket

Verification

  • Retrieve lifecycle configuration for the bucket:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api get-bucket-lifecycle-configuration --bucket BUCKET_NAME

    Example

    [user@client ~]# aws --endpoint-url=http://host01:80  s3api get-bucket-lifecycle-configuration --bucket testbucket

  • Optional: From the Ceph Object Gateway node, retrieve the bucket lifecycle configuration:

    Syntax

    radosgw-admin lc get --bucket=BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin lc get --bucket=testbucket

    Note

    The command does not return any information if a bucket lifecycle policy is not present.

Additional Resources

  • See the S3 bucket lifecycle section in the Red Hat Ceph Storage Developer Guide for details.

3.10.3. Updating a lifecycle management policy

You can update a lifecycle management policy by using the s3cmd put-bucket-lifecycle-configuration command.

Note

The put-bucket-lifecycle-configuration overwrites an existing bucket lifecycle configuration. If you want to retain any of the current lifecycle policy settings, you must include them in the lifecycle configuration file.

Prerequisites

  • A running Red Hat Storage cluster.
  • Installation of the Ceph Object Gateway.
  • Root-level access to a Ceph Object Gateway node.
  • An S3 bucket created.
  • An S3 user created with user access.
  • Access to a Ceph Object Gateway client with the AWS CLI package installed.

Procedure

  1. Create a JSON file for the lifecycle configuration:

    Example

    [user@client ~]$ vi lifecycle.json

  2. Add the specific lifecycle configuration rules to the file:

    Example

    {
    	"Rules": [
            {
    		    "Filter": {
    			    "Prefix": "images/"
    		    },
    		    "Status": "Enabled",
    		    "Expiration": {
    			    "Days": 1
    		    },
    		    "ID": "ImageExpiration"
    	    },
    		{
    			"Filter": {
    				"Prefix": "docs/"
    			},
    			"Status": "Enabled",
    			"Expiration": {
    				"Days": 30
    			},
    			"ID": "DocsExpiration"
    		}
    	]
    }

  3. Update the lifecycle configuration on the bucket:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

Verification

  • Retrieve the lifecycle configuration for the bucket:

    Syntax

    aws --endpointurl=RADOSGW_ENDPOINT_URL:PORT s3api get-bucket-lifecycle-configuration --bucket BUCKET_NAME

    Example

    [user@client ~]$ aws -endpoint-url=http://host01:80 s3api get-bucket-lifecycle-configuration --bucket testbucket
    
    {
        "Rules": [
            {
                "Expiration": {
                    "Days": 30
                },
                "ID": "DocsExpiration",
                "Filter": {
                    "Prefix": "docs/"
                },
                "Status": "Enabled"
            },
            {
                "Expiration": {
                    "Days": 1
                },
                "ID": "ImageExpiration",
                "Filter": {
                    "Prefix": "images/"
                },
                "Status": "Enabled"
            }
        ]
    }

  • Optional: From the Ceph Object Gateway node, log into the Cephadm shell and retrieve the bucket lifecycle configuration:

    Syntax

    radosgw-admin lc get --bucket=BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin lc get --bucket=testbucket
    {
    	"prefix_map": {
            "docs/": {
    			"status": true,
    			"dm_expiration": false,
    			"expiration": 1,
    			"noncur_expiration": 0,
    			"mp_expiration": 0,
    			"transitions": {},
    			"noncur_transitions": {}
    		},
    		"images/": {
    			"status": true,
    			"dm_expiration": false,
    			"expiration": 1,
    			"noncur_expiration": 0,
    			"mp_expiration": 0,
    			"transitions": {},
    			"noncur_transitions": {}
    		}
    	},
    	"rule_map": [
            {
            "id": "DocsExpiration",
        	"rule": {
        		"id": "DocsExpiration",
        		"prefix": "",
        		"status": "Enabled",
        		"expiration": {
        			"days": "30",
        			"date": ""
        		},
                "noncur_expiration": {
                    "days": "",
                    "date": ""
                },
        		"mp_expiration": {
        			"days": "",
        			"date": ""
        		},
        		"filter": {
        			"prefix": "docs/",
        			"obj_tags": {
        				"tagset": {}
        			}
        		},
        		"transitions": {},
        		"noncur_transitions": {},
        		"dm_expiration": false
        	}
        },
        {
    		"id": "ImageExpiration",
    		"rule": {
    			"id": "ImageExpiration",
    			"prefix": "",
    			"status": "Enabled",
    			"expiration": {
    				"days": "1",
    				"date": ""
    			},
    			"mp_expiration": {
    				"days": "",
    				"date": ""
    			},
    			"filter": {
    				"prefix": "images/",
    				"obj_tags": {
    					"tagset": {}
    				}
    			},
    			"transitions": {},
    			"noncur_transitions": {},
    			"dm_expiration": false
    		}
    	}
      ]
    }

Additional Resources

3.10.4. Monitoring bucket lifecycles

You can monitor lifecycle processing and manually process the lifecycle of buckets with the radosgw-admin lc list and radosgw-admin lc process commands.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to a Ceph Object Gateway node.
  • Creation of an S3 bucket with a lifecycle configuration policy applied.

Procedure

  1. List bucket lifecycle progress:

    Example

    [root@rgw ~]# radosgw-admin lc list
    
    [
       {
             “bucket”: “:testbucket:8b63d584-9ea1-4cf3-8443-a6a15beca943.54187.1”,
             “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
             “status” : “UNINITIAL”
       },
       {
             “bucket”: “:testbucket1:8b635499-9e41-4cf3-8443-a6a15345943.54187.2”,
             “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
             “status” : “UNINITIAL”
       }
    ]

    The bucket lifecycle processing status can be one of the following:

    • UNINITIAL - The process has not run yet.
    • PROCESSING - The process is currently running.
    • COMPLETE - The process has completed.
  2. Optional: You can manually process bucket lifecycle policies:

    1. Process the lifecycle policy for a single bucket:

      Syntax

      radosgw-admin lc process --bucket=BUCKET_NAME

      Example

      [root@rgw ~]# radosgw-admin lc process --bucket=testbucket1

    2. Process all bucket lifecycle policies immediately:

      Example

      [root@rgw ~]# radosgw-admin lc process

Verification

  • List the bucket lifecycle policies:

    [root@rgw ~]# radosgw-admin lc list
    [
        {
              “bucket”: “:testbucket:8b63d584-9ea1-4cf3-8443-a6a15beca943.54187.1”,
              “started”: “Thu, 17 Mar 2022 21:48:50 GMT”,
              “status” : “COMPLETE”
        }
        {
              “bucket”: “:testbucket1:8b635499-9e41-4cf3-8443-a6a15345943.54187.2”,
              “started”: “Thu, 17 Mar 2022 20:38:50 GMT”,
              “status” : “COMPLETE”
        }
    ]

Additional Resources

  • See the S3 bucket lifecycle section in the Red Hat Ceph Storage Developer Guide for details.

3.10.5. Configuring lifecycle expiration window

You can set the time that the lifecycle management process runs each day by setting the rgw_lifecycle_work_time parameter. By default, lifecycle processing occurs once per day, at midnight.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway.
  • Root-level access to a Ceph Object Gateway node.

Procedure

  1. Set the lifecycle expiration time:

    Syntax

    ceph config set client.rgw rgw_lifecycle_work_time %D:%D-%D:%D

    Replace %d:%d-%d:%d with start_hour:start_minute-end_hour:end_minute.

    Example

    [root@rgw ~]# ceph config set client.rgw rgw_lifecycle_work_time 06:00-08:00

Verification

  • Retrieve the lifecycle expiration work time:

    Example

    [root@rgw ~]# ceph config get client.rgw rgw_lifecycle_work_time
    
    06:00-08:00

Additional Resources

  • See the S3 bucket lifecycle section in the Red Hat Ceph Storage Developer Guide for details.

3.10.6. S3 bucket lifecycle transition within a storage cluster

You can use a bucket lifecycle configuration to manage objects so objects are stored effectively throughout the object’s lifetime. The object lifecycle transition rule allows you to manage,and effectively store the objects throughout the object’s lifetime. You can transition objects to less expensive storage classes,archive or even delete them.

You can create storage classes for:

  • Fast media, such as, SSD or NVMe for I/O sensitive workloads
  • Slow magnetic media, such as, SAS or SATA for archiving.

You can create a schedule for data movement between a hot storage class and a cold storage class. You can schedule this movement after a specified time so that the object expires and is deleted permanently for example you can transition objects to a storage class 30 days after you have created or even archived the objects to a storage class one year after creating them. You can do this through a transition rule. This rule applies to an object transitioning from one storage class to another. The lifecycle configuration contains one or more rules using the <Rule> element.

Additional Resources

  • See the Red Hat Ceph Storage Developer Guide for details on bucket lifecycle.

3.10.7. Transitioning an object from one storage class to another

The object lifecycle transition rule allows you to transition an object from one storage class to another class.

Prerequisites

  • Installation of the Ceph Object Gateway software.
  • Root-level access to the Ceph Object Gateway node.
  • An S3 user created with user access.

Procedure

  1. Create a new data pool:

    Syntax

    ceph osd pool create POOL_NAME

    Example

    [root@rgw ~]# ceph osd pool create test.hot.data

  2. Add a new storage class:

    Syntax

    radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

    Example

    [root@rgw ~]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class hot.test
    {
            "key": "default-placement",
            "val": {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "STANDARD",
                    "hot.test"
                ]
            }
        }

  3. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [root@rgw ~]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class hot.test --data-pool test.hot.data
    {
               "key": "default-placement",
               "val": {
                   "index_pool": "test_zone.rgw.buckets.index",
                   "storage_classes": {
                       "STANDARD": {
                           "data_pool": "test.hot.data"
                       },
                       "hot.test": {
                           "data_pool": "test.hot.data",
                      }
                   },
                   "data_extra_pool": "",
                   "index_type": 0
               }

    Note

    Consider setting the compression_type when creating cold or archival data storage pools with write once.

  4. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [root@rgw ~] ceph osd pool application enable test.hot.data rgw
    enabled application 'rgw' on pool 'test.hot.data'

  5. Restart all the rgw daemons.
  6. Create a bucket:

    Example

    [root@rgw ~]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

  7. Add the object:

    Example

    [root@rgw ~]# aws --endpoint=http://1x.7x.2xx.1xx:80 s3api put-object --bucket testbucket10  --key compliance-upload --body /root/test2.txt

  8. Create a second data pool:

    Syntax

    ceph osd pool create POOL_NAME

    Example

    [root@rgw ~]# ceph osd pool create test.cold.data

  9. Add a new storage class:

    Syntax

    radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

    Example

    [root@rgw ~]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class cold.test
    {
            "key": "default-placement",
            "val": {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "STANDARD",
                    "cold.test"
                ]
            }
        }

  10. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [root@rgw ~]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class cold.test --data-pool test.cold.data

  11. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [root@rgw ~] ceph osd pool application enable test.cold.data rgw
    enabled application 'rgw' on pool 'test.cold.data'

  12. Restart all the rgw daemons.
  13. To view zone group configuration, execute:

    Syntax

    radosgw-admin zonegroup get
    {
        "id": "3019de59-ddde-4c5c-b532-7cdd29de09a1",
        "name": "default",
        "api_name": "default",
        "is_master": "true",
        "endpoints": [],
        "hostnames": [],
        "hostnames_s3website": [],
        "master_zone": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "zones": [
            {
                "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
                "name": "default",
                "endpoints": [],
                "log_meta": "false",
                "log_data": "false",
                "bucket_index_max_shards": 11,
                "read_only": "false",
                "tier_type": "",
                "sync_from_all": "true",
                "sync_from": [],
                "redirect_zone": ""
            }
        ],
        "placement_targets": [
            {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "hot.test",
                    "cold.test",
                    "STANDARD"
                ]
            }
        ],
        "default_placement": "default-placement",
        "realm_id": "",
        "sync_policy": {
            "groups": []
        }
    }

  14. To view the zone configuration,execute:

    Syntax

    radosgw-admin zone get
    {
        "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "name": "default",
        "domain_root": "default.rgw.meta:root",
        "control_pool": "default.rgw.control",
        "gc_pool": "default.rgw.log:gc",
        "lc_pool": "default.rgw.log:lc",
        "log_pool": "default.rgw.log",
        "intent_log_pool": "default.rgw.log:intent",
        "usage_log_pool": "default.rgw.log:usage",
        "roles_pool": "default.rgw.meta:roles",
        "reshard_pool": "default.rgw.log:reshard",
        "user_keys_pool": "default.rgw.meta:users.keys",
        "user_email_pool": "default.rgw.meta:users.email",
        "user_swift_pool": "default.rgw.meta:users.swift",
        "user_uid_pool": "default.rgw.meta:users.uid",
        "otp_pool": "default.rgw.otp",
        "system_key": {
            "access_key": "",
            "secret_key": ""
        },
        "placement_pools": [
            {
                "key": "default-placement",
                "val": {
                    "index_pool": "default.rgw.buckets.index",
                    "storage_classes": {
                        "cold.test": {
                            "data_pool": "test.cold.data"
                        },
                        "hot.test": {
                            "data_pool": "test.hot.data"
                        },
                        "STANDARD": {
                            "data_pool": "default.rgw.buckets.data"
                        }
                    },
                    "data_extra_pool": "default.rgw.buckets.non-ec",
                    "index_type": 0
                }
            }
        ],
        "realm_id": "",
        "notif_pool": "default.rgw.log:notif"
    }

  15. Create a bucket:

    Example

    [root@rgw ~]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

  16. Create a JSON file for lifecycle configuration:

    Example

    [root@rgw ~]# vi lifecycle.json

  17. Add the specific lifecyle configuration rule in the file:

    Example

    {
        "Rules": [
            {
                "Filter": {
                    "Prefix": ""
                },
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    },
     {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    }
                ],
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration"
            }
        ]
    }

    The lifecycle configuration example shows an object that will transition from the default STANDARD storage class to the hot.test storage class after 5 days, again transitions after 20 days to the cold.test storage class, and finally expires after 365 days in the cold.test storage class.

  18. Set the lifecycle configuration on the bucket:

    Example

    [root@rgw ~] aws s3api put-bucket-lifecycle-configuration --bucket testbucket20 --lifecycle-configuration file://lifecycle.json

  19. Retrieve the lifecycle configuration on the bucket:

    Example

    [root@rgw ~]aws s3api get-bucket-lifecycle-configuration --bucket testbucket20
    {
        "Rules": [
            {
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration",
                "Prefix": "",
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    },
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    }
                ]
            }
        ]
    }

Additional Resources

  • See the Red Hat Ceph Storage Developer Guide for details on bucket lifecycle.

3.11. Ceph Object Gateway data layout

Although RADOS only knows about pools and objects with their Extended Attributes (xattrs) and object map (OMAP), conceptually Ceph Object Gateway organizes its data into three different kinds:

  • metadata
  • bucket index
  • data

Metadata

There are three sections of metadata:

  • user: Holds user information.
  • bucket: Holds a mapping between bucket name and bucket instance ID.
  • bucket.instance: Holds bucket instance information.

You can use the following commands to view metadata entries:

Syntax

radosgw-admin metadata get bucket:BUCKET_NAME
radosgw-admin metadata get bucket.instance:BUCKET:BUCKET_ID
radosgw-admin metadata get user:USER
radosgw-admin metadata set user:USER

Example

[root@host01 ~]# radosgw-admin metadata list
[root@host01 ~]# radosgw-admin metadata list bucket
[root@host01 ~]# radosgw-admin metadata list bucket.instance
[root@host01 ~]# radosgw-admin metadata list user

Every metadata entry is kept on a single RADOS object.

Note

A Ceph Object Gateway object might consist of several RADOS objects, the first of which is the head that contains the metadata, such as manifest, Access Control List (ACL), content type, ETag, and user-defined metadata. The metadata is stored in xattrs. The head might also contain up to 512 KB of object data, for efficiency and atomicity. The manifest describes how each object is laid out in RADOS objects.

Bucket index

It is a different kind of metadata, and kept separately. The bucket index holds a key-value map in RADOS objects. By default, it is a single RADOS object per bucket, but it is possible to shard the map over multiple RADOS objects.

The map itself is kept in OMAP associated with each RADOS object. The key of each OMAP is the name of the objects, and the value holds some basic metadata of that object, the metadata that appears when listing the bucket. Each OMAP holds a header, and we keep some bucket accounting metadata in that header such as number of objects, total size, and the like.

Note

OMAP is a key-value store, associated with an object, in a way similar to how extended attributes associate with a POSIX file. An object’s OMAP is not physically located in the object’s storage, but its precise implementation is invisible and immaterial to the Ceph Object Gateway.

Data

Objects data is kept in one or more RADOS objects for each Ceph Object Gateway object.

3.11.1. Object lookup path

When accessing objects, REST APIs come to Ceph Object Gateway with three parameters:

  • Account information, which has the access key in S3 or account name in Swift
  • Bucket or container name
  • Object name or key

At present, Ceph Object Gateway only uses account information to find out the user ID and for access control. It uses only the bucket name and object key to address the object in a pool.

Account information

The user ID in Ceph Object Gateway is a string, typically the actual user name from the user credentials and not a hashed or mapped identifier.

When accessing a user’s data, the user record is loaded from an object USER_ID in the default.rgw.meta pool with users.uid namespace.

Bucket names

They are represented in the default.rgw.meta pool with root namespace. Bucket record is loaded in order to obtain a marker, which serves as a bucket ID.

Object names

The object is located in the default.rgw.buckets.data pool. Object name is MARKER_KEY, for example default.7593.4_image.png, where the marker is default.7593.4 and the key is image.png. These concatenated names are not parsed and are passed down to RADOS only. Therefore, the choice of the separator is not important and causes no ambiguity. For the same reason, slashes are permitted in object names, such as keys.

3.11.1.1. Multiple data pools

It is possible to create multiple data pools so that different users’ buckets are created in different RADOS pools by default, thus providing the necessary scaling. The layout and naming of these pools is controlled by a policy setting.

3.11.2. Bucket and object listing

Buckets that belong to a given user are listed in an OMAP of an object named USER_ID.buckets, for example, foo.buckets, in the default.rgw.meta pool with users.uid namespace. These objects are accessed when listing buckets, when updating bucket contents, and updating and retrieving bucket statistics such as quota. These listings are kept consistent with buckets in the .rgw pool.

Note

See the user-visible, encoded class cls_user_bucket_entry and its nested class cls_user_bucket for the values of these OMAP entries.

Objects that belong to a given bucket are listed in a bucket index. The default naming for index objects is .dir.MARKER in the default.rgw.buckets.index pool.

Additional Resources

3.12. Object Gateway data layout parameters

This is a list of data layout parameters for Ceph Object Gateway.

Known pools:

.rgw.root
Unspecified region, zone, and global information records, one per object.
ZONE.rgw.control
notify.N
ZONE.rgw.meta

Multiple namespaces with different kinds of metadata

namespace: root

BUCKET .bucket.meta.BUCKET:MARKER # see put_bucket_instance_info()

The tenant is used to disambiguate buckets, but not bucket instances.

Example

.bucket.meta.prodtx:test%25star:default.84099.6
.bucket.meta.testcont:default.4126.1
.bucket.meta.prodtx:testcont:default.84099.4
prodtx/testcont
prodtx/test%25star
testcont

namespace: users.uid

Contains _both_ per-user information (RGWUserInfo) in USER objects and per-user lists of buckets in omaps of USER.buckets objects. The USER might contain the tenant if non-empty.

Example

prodtx$prodt
test2.buckets
prodtx$prodt.buckets
test2

namespace: users.email
Unimportant
namespace: users.keys

47UA98JSTJZ9YAN3OS3O

This allows Ceph Object Gateway to look up users by their access keys during authentication.

namespace: users.swift
test:tester
ZONE.rgw.buckets.index
Objects are named .dir.MARKER, each contains a bucket index. If the index is sharded, each shard appends the shard index after the marker.
ZONE.rgw.buckets.data

default.7593.4__shadow_.488urDFerTYXavx4yAd-Op8mxehnvTI_1 MARKER_KEY

An example of a marker would be default.16004.1 or default.7593.4. The current format is ZONE.INSTANCE_ID.BUCKET_ID, but once generated, a marker is not parsed again, so its format might change freely in the future.

Additional Resources

3.13. Session tags for Attribute-based access control (ABAC) in STS

Session tags are key-value pairs that can be passed while federating a user. They are passed as aws:PrincipalTag in the session or temporary credentials that are returned back by secure token service (STS). These principal tags consist of session tags that come in as part of the web token and tags that are attached to the role being assumed.

Note

Currently, the session tags are only supported as part of the web token passed to AssumeRoleWithWebIdentity.

The tags have to be always specified in the following namespace: https://aws.amazon.com/tags.

Important

The trust policy must have sts:TagSession permission if the web token passed in by the federated user contains session tags. Otherwise, the AssumeRoleWithWebIdentity action fails.

Example of the trust policy with sts:TagSession:

{
        "Version":"2012-10-17",
        "Statement":[
        {
            "Effect":"Allow",
            "Action":["sts:AssumeRoleWithWebIdentity","sts:TagSession"],
            "Principal":{"Federated":["arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/quickstart"]},
            "Condition":{"StringEquals":{"localhost:8080/auth/realms/quickstart:sub":"test"}}
        }]
    }

Properties

The following are the properties of session tags:

  • Session tags can be multi-valued.

    Note

    Multi-valued session tags are not supported in Amazon Web Service (AWS).

  • Keycloak can be set up as an OpenID Connect Identity Provider (IDP) with a maximum of 50 session tags.
  • The maximum size of a key allowed is 128 characters.
  • The maximum size of a value allowed is 256 characters.
  • The tag or the value cannot start with aws:.

Additional Resources

3.13.1. Tag keys

The following are the tag keys that can be used in the role trust policy or the role permission policy.

aws:RequestTag
Description

Compares the key-value pair passed in the request with the key-value pair in the role’s trust policy.

In the case of AssumeRoleWithWebIdentity, session tags can be used as aws:RequestTag in the role trust policy. Those session tags are passed by Keycloak in the web token. As a result, a federated user can assume a role.

aws:PrincipalTag
Description

Compares the key-value pair attached to the principal with the key-value pair in the policy.

In the case of AssumeRoleWithWebIdentity, session tags appear as principal tags in the temporary credentials once a user is authenticated. Those session tags are passed by Keycloak in the web token. They can be used as aws:PrincipalTag in the role permission policy.

iam:ResourceTag
Description

Compares the key-value pair attached to the resource with the key-value pair in the policy.

In the case of AssumeRoleWithWebIdentity, tags attached to the role are compared with those in the trust policy to allow a user to assume a role.

Note

The Ceph Object Gateway now supports RESTful APIs for tagging, listing tags, and untagging actions on a role.

aws:TagKeys
Description

Compares tags in the request with the tags in the policy.

In the case of AssumeRoleWithWebIdentity, tags are used to check the tag keys in a role trust policy or permission policy before a user is allowed to assume a role.

s3:ResourceTag
Description

Compares tags present on the S3 resource, that is bucket or object, with the tags in the role’s permission policy.

It can be used for authorizing an S3 operation in the Ceph Object Gateway. However, this is not allowed in AWS.

It is a key used to refer to tags that have been attached to an object or a bucket. Tags can be attached to an object or a bucket using RESTful APIs available for the same.

3.13.2. S3 resource tags

The following list shows which S3 resource tag type is supported for authorizing a particular operation.

Tag type: Object tags
Operations
GetObject, GetObjectTags, DeleteObjectTags, DeleteObject, PutACLs, InitMultipart, AbortMultipart , `ListMultipart, GetAttrs, PutObjectRetention, GetObjectRetention, PutObjectLegalHold, GetObjectLegalHold
Tag type: Bucket tags
Operations
PutObjectTags, GetBucketTags, PutBucketTags, DeleteBucketTags, GetBucketReplication, DeleteBucketReplication, GetBucketVersioning, SetBucketVersioning, GetBucketWebsite, SetBucketWebsite, DeleteBucketWebsite, StatBucket, ListBucket, GetBucketLogging, GetBucketLocation, DeleteBucket, GetLC, PutLC, DeleteLC, GetCORS, PutCORS, GetRequestPayment, SetRequestPayment. PutBucketPolicy, GetBucketPolicy, DeleteBucketPolicy, PutBucketObjectLock, GetBucketObjectLock, GetBucketPolicyStatus, PutBucketPublicAccessBlock, GetBucketPublicAccessBlock, DeleteBucketPublicAccessBlock
Tag type: Bucket tags for bucket ACLs, Object tags for object ACLs
Operations
GetACLs, PutACLs
Tag type: Object tags of source object, Bucket tags of destination bucket
Operations
PutObject, CopyObject

3.14. Optimize the Ceph Object Gateway’s garbage collection

When new data objects are written into the storage cluster, the Ceph Object Gateway immediately allocates the storage for these new objects. After you delete or overwrite data objects in the storage cluster, the Ceph Object Gateway deletes those objects from the bucket index. Some time afterward, the Ceph Object Gateway then purges the space that was used to store the objects in the storage cluster. The process of purging the deleted object data from the storage cluster is known as Garbage Collection, or GC.

Garbage collection operations typically run in the background. You can configure these operations to either execute continuously, or to run only during intervals of low activity and light workloads. By default, the Ceph Object Gateway conducts GC operations continuously. Because GC operations are a normal part of Ceph Object Gateway operations, deleted objects that are eligible for garbage collection exist most of the time.

3.14.1. Viewing the garbage collection queue

Before you purge deleted and overwritten objects from the storage cluster, use radosgw-admin to view the objects awaiting garbage collection.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the Ceph Object Gateway.

Procedure

  1. To view the queue of objects awaiting garbage collection:

    Example

    [root@rgw ~] radosgw-admin gc list

Note

To list all entries in the queue, including unexpired entries, use the --include-all option.

3.14.2. Adjusting garbage collection for delete-heavy workloads

Some workloads may temporarily or permanently outpace the rate of garbage collection activity. This is especially true of delete-heavy workloads, where many objects get stored for a short period of time and are then deleted. For these types of workloads, consider increasing the priority of garbage collection operations relative to other operations. Contact Red Hat Support with any additional questions about Ceph Object Gateway Garbage Collection.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all nodes in the storage cluster.

Procedure

  1. Open /etc/ceph/ceph.conf for editing.
  2. Set the value of rgw_gc_max_concurrent_io to 20, and the value of rgw_gc_max_trim_chunk to 64.

    rgw_gc_max_concurrent_io = 20
    rgw_gc_max_trim_chunk = 64
  3. Restart the Ceph Object Gateway to allow the changed settings to take effect.
  4. Monitor the storage cluster during GC activity to verify that the increased values do not adversely affect performance.
Important

Never modify the value for the rgw_gc_max_objs option in a running cluster. You should only change this value before deploying the RGW nodes.

3.14.3. Viewing the number of objects garbage collected

You can view the number of objects garbage collected since the restart of the Ceph Object Gateway using gc_retire_object parameter in the performance dump.

This counter can be monitored to determine the delta between the number of objects at a specific time frame for that Ceph Object Gateway, for example, the average number of objects garbage collected weekly, daily, or hourly. This can be used to determine how efficiently garbage collection progresses per Ceph Object Gateway.

The socket file is located in the /var/run/ceph directory.

Prerequisites

  • A running Red Hat Ceph Storage cluster with Ceph Object Gateway installed.
  • Root-level access to all nodes in the storage cluster.
  • The ceph-common packages installed on the Ceph Monitor.

Procedure

  • Access the performance counter data using grep for the gc_retire_object counter:

    Syntax

    ceph --admin-daemon PATH_TO_SOCKET_FILE perf dump | grep gc_retire_object

    Example

    [root@mon ~]# ceph --admin-daemon /var/run/ceph/ceph-client.rgw.f27-h25-000-6048r.rgw0.104.93991732704816.asok perf dump | grep gc_retire_object

Additional Resources

  • See the Red Hat Knowledgebase article Ceph RGW - GC Tuning Options for more information.
  • See the General Settings section in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide for more information.
  • See the Configuration Reference section in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide for more information.
  • See the Ceph Object Gateway metrics section in the Red Hat Ceph Storage Administration Guide for more information.

3.15. Optimize the Ceph Object Gateway’s data object storage

Bucket life cycle configuration optimizes data object storage to increase its efficiency and to provide effective storage throughout the lifetime of the data.

The S3 API in the Ceph Object Gateway currently supports a subset of the AWS bucket life cycle configuration actions:

  • Expiration
  • NoncurrentVersionExpiration
  • AbortIncompleteMultipartUpload

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all of the nodes in the storage cluster.

3.15.1. Parallel thread processing for bucket life cycles

The Ceph Object Gateway now allows for parallel thread processing of bucket life cycles across multiple Ceph Object Gateway instances. Increasing the number of threads that run in parallel enables the Ceph Object Gateway to process large workloads more efficiently. In addition, the Ceph Object Gateway now uses a numbered sequence for index shard enumeration instead of using in-order numbering.

3.15.2. Optimizing the bucket life cycle

Two options in the Ceph configuration file affect the efficiency of bucket life cycle processing:

  • rgw_lc_max_worker specifies the number of life cycle worker threads to run in parallel. This enables the simultaneous processing of both bucket and index shards. The default value for this option is 3.
  • rgw_lc_max_wp_worker specifies the number of threads in each life cycle worker thread’s work pool. This option helps to accelerate processing for each bucket. The default value for this option is 3.

For a workload with a large number of buckets — for example, a workload with thousands of buckets — consider increasing the value of the rgw_lc_max_worker option.

For a workload with a smaller number of buckets but with a higher number of objects in each bucket — such as in the hundreds of thousands — consider increasing the value of the rgw_lc_max_wp_worker option.

Note

Before increasing the value of either of these options, please validate current storage cluster performance and Ceph Object Gateway utilization. Red Hat does not recommend that you assign a value of 10 or above for either of these options.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all of the nodes in the storage cluster.

Procedure

  1. Open /etc/ceph/ceph.conf for editing.
  2. To increase the number of threads to run in parallel, set the value of rgw_lc_max_worker to a value between 3 and 9:

    Syntax

    rgw_lc_max_worker = VALUE

    Example

    rgw_lc_max_worker = 7

  3. To increase the number of threads in each thread’s work pool, set the value of rgw_lc_max_wp_worker to a value between 3 and 9:

    Syntax

    rgw_lc_max_wp_worker = VALUE

    Example

    rgw_lc_max_wp_worker = 7

  4. Restart the Ceph Object Gateway to allow the changed settings to take effect.
  5. Monitor the storage cluster to verify that the increased values do not adversely affect performance.

Additional Resources

3.15.3. Additional Resources

3.16. The Ceph Object Gateway and multi-factor authentication

As a storage administrator, you can manage time-based one time password (TOTP) tokens for Ceph Object Gateway users.

3.16.1. Multi-factor authentication

When a bucket is configured for object versioning, you can optionally configure the bucket to require multi-factor authentication (MFA) for delete requests. Using MFA, a time-based one time password (TOTP) token is passed as a key to the x-amz-mfa header. The tokens are generated with virtual MFA devices like Google Authenticator, or a hardware MFA device like those provided by Gemalto.

Use radosgw-admin to assign time-based one time password tokens to a user. You must set a secret seed and a serial ID. You can also use radosgw-admin to list, remove, and resynchronize tokens.

Important

In a multisite environment it is advisable to use different tokens for different zones, because, while MFA IDs are set on the user’s metadata, the actual MFA one time password configuration resides on the local zone’s OSDs.

Table 3.1. Terminology
TermDescription

TOTP

Time-based One Time Password.

Token serial

A string that represents the ID of a TOTP token.

Token seed

The secret seed that is used to calculate the TOTP. It can be hexadecimal or base32.

TOTP seconds

The time resolution used for TOTP generation.

TOTP window

The number of TOTP tokens that are checked before and after the current token when validating tokens.

TOTP pin

The valid value of a TOTP token at a certain time.

3.16.2. Creating a seed for multi-factor authentication

To set up multi-factor authentication (MFA), you must create a secret seed for use by the one-time password generator and the back-end MFA system.

Prerequisites

  • A Linux system.
  • Access to the command line shell.

Procedure

  1. Generate a 30 character seed from the urandom Linux device file and store it in the shell variable SEED:

    Example

    [user@host ~]$ SEED=$(head -10 /dev/urandom | sha512sum | cut -b 1-30)

  2. Print the seed by running echo on the SEED variable:

    Example

    [user@host ~]$ echo $SEED
    492dedb20cf51d1405ef6a1316017e

    Configure the one-time password generator and the back-end MFA system to use the same seed.

Additional Resources

3.16.3. Creating a new multi-factor authentication TOTP token

Create a new multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • A secret seed for the one-time password generator and Ceph Object Gateway MFA was generated.

Procedure

  1. Create a new MFA TOTP token:

    Syntax

    radosgw-admin mfa create --uid=USERID --totp-serial=SERIAL --totp-seed=SEED --totp-seed-type=SEED_TYPE --totp-seconds=TOTP_SECONDS --totp-window=TOTP_WINDOW

    Set USERID to the user name to set up MFA on, set SERIAL to a string that represents the ID for the TOTP token, and set SEED to a hexadecimal or base32 value that is used to calculate the TOTP. The following settings are optional: Set the SEED_TYPE to hex or base32, set TOTP_SECONDS to the timeout in seconds, or set TOTP_WINDOW to the number of TOTP tokens to check before and after the current token when validating tokens.

    Example

    [root@mon ~]# radosgw-admin mfa create --uid=johndoe --totp-serial=MFAtest --totp-seed=492dedb20cf51d1405ef6a1316017e

Additional Resources

3.16.4. Test a multi-factor authentication TOTP token

Test a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Test the TOTP token PIN to verify that TOTP functions correctly:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the latest PIN from the one-time password generator.

    Example

    [root@mon ~] # radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

    If this is the first time you have tested the PIN, it may fail. If it fails, resynchronize the token. See Resynchronizing a multi-factor authentication token in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

Additional Resources

3.16.5. Resynchronizing a multi-factor authentication TOTP token

Resynchronize a multi-factor authentication (MFA) time-based one time password token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Resynchronize a multi-factor authentication TOTP token in case of time skew or failed checks.

    This requires passing in two consecutive pins: the previous pin, and the current pin.

    Syntax

    radosgw-admin mfa resync --uid=USERID --totp-serial=SERIAL --totp-pin=PREVIOUS_PIN --totp=pin=CURRENT_PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, set PREVIOUS_PIN to the user’s previous PIN, and set CURRENT_PIN to the user’s current PIN.

    Example

    radosgw-admin mfa resync --uid=johndoe --totp-serial=MFAtest --totp-pin=802017 --totp-pin=439996

  2. Verify the token was successfully resynchronized by testing a new PIN:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the user’s PIN.

    Example

    [root@mon ~]# radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

Additional Resources

3.16.6. Listing multi-factor authentication TOTP tokens

List all multi-factor authentication (MFA) time-based one time password (TOTP) tokens that a particular user has.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. List MFA TOTP tokens:

    Syntax

    radosgw-admin mfa list --uid=USERID

    Set USERID to the user name MFA is set up on.

    Example

    [root@mon ~]# radosgw-admin mfa list --uid=johndoe
    {
        "entries": [
            {
                "type": 2,
                "id": "MFAtest",
                "seed": "492dedb20cf51d1405ef6a1316017e",
                "seed_type": "hex",
                "time_ofs": 0,
                "step_size": 30,
                "window": 2
            }
        ]
    }

Additional Resources

3.16.7. Display a multi-factor authentication TOTP token

Display a specific multi-factor authentication (MFA) time-based one time password (TOTP) token by specifying its serial.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Show the MFA TOTP token:

    Syntax

    radosgw-admin mfa get --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

Additional Resources

3.16.8. Deleting a multi-factor authentication TOTP token

Delete a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Delete an MFA TOTP token:

    Syntax

    radosgw-admin mfa remove --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@mon ~]# radosgw-admin mfa remove --uid=johndoe --totp-serial=MFAtest

  2. Verify the MFA TOTP token was deleted:

    Syntax

    radosgw-admin mfa get --uid=_USERID_ --totp-serial=_SERIAL_

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@mon ~]# radosgw-admin mfa get --uid=johndoe --totp-serial=MFAtest
    MFA serial id not found

Additional Resources

3.17. Removing Ceph Object Gateway using Ansible

To remove a Ceph Object Gateway with Ansible, use the shrink-rgw.yml playbook.

Prerequisites

  • An Ansible administration node.
  • A running Red Hat Ceph Storage cluster deployed by Ansible.

Procedure

  1. Change to the /usr/share/ceph-ansible/ directory.

    [user@admin ~]$ cd /usr/share/ceph-ansible
  2. For bare-metal and containers deployments, run the shrink-rgw.yml Ansible playbook:

    Syntax

    ansible-playbook infrastructure-playbooks/shrink-rgw.yml -e rgw_to_kill=HOSTNAME.rgw_INSTANCE_NAME_ -u ANSIBLE_USER_NAME -i hosts

    Replace:

    • HOSTNAME with the short host name of the Ceph Object Gateway node. You can remove only one Ceph Object Gateway each time the playbook runs.
    • ANSIBLE_USER_NAME with the name of the Ansible user

    Example

    [user@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/shrink-rgw.yml -e rgw_to_kill=node03.rgw0 -u admin -i hosts

  3. Remove the Ceph Object Gateway entry from all Ceph configuration files in the storage cluster.
  4. Ensure that the Ceph Object Gateway has been successfully removed:

    [root@mon ~]# ceph -s

Additional Resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.