Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 10. Management of Ceph object gateway using the Ceph Orchestrator

As a storage administrator, you can deploy Ceph object gateway using the command line interface or by using the service specification.

You can also configure multi-site object gateways, and remove the Ceph object gateway using the Ceph Orchestrator.

Cephadm deploys Ceph object gateway as a collection of daemons that manages a single-cluster deployment or a particular realm and zone in a multisite deployment.

Note

With Cephadm, the object gateway daemons are configured using the monitor configuration database instead of a ceph.conf or the command line. If that configuration is not already in the client.rgw section, then the object gateway daemons will start up with default settings and bind to the port 80.

Note

The .default.rgw.buckets.index pool is created only after the bucket is created in Ceph Object Gateway, while the .default.rgw.buckets.data pool is created after the data is uploaded to the bucket.

This section covers the following administrative tasks:

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • All the managers, monitors, and OSDs are deployed in the storage cluster.

Using the Ceph Orchestrator, you can deploy the Ceph Object Gateway with the ceph orch command in the command line interface.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • All manager, monitor and OSD daemons are deployed.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. You can deploy the Ceph object gateway daemons in three different ways:

Method 1

  • Create realm, zone group, zone, and then use the placement specification with the host name:

    1. Create a realm:

      Syntax

      radosgw-admin realm create --rgw-realm=REALM_NAME --default
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default
      Copy to Clipboard Toggle word wrap

    2. Create a zone group:

      Syntax

      radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME  --master --default
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=default  --master --default
      Copy to Clipboard Toggle word wrap

    3. Create a zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=test_zone --master --default
      Copy to Clipboard Toggle word wrap

    4. Commit the changes:

      Syntax

      radosgw-admin period update --rgw-realm=REALM_NAME --commit
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin period update --rgw-realm=test_realm --commit
      Copy to Clipboard Toggle word wrap

    5. Run the ceph orch apply command:

      Syntax

      ceph orch apply rgw NAME [--realm=REALM_NAME] [--zone=ZONE_NAME] --placement="NUMBER_OF_DAEMONS [HOST_NAME_1 HOST_NAME_2]"
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph orch apply rgw test --realm=test_realm --zone=test_zone --placement="2 host01 host02"
      Copy to Clipboard Toggle word wrap

Method 2

  • Use an arbitrary service name to deploy two Ceph Object Gateway daemons for a single cluster deployment:

    Syntax

    ceph orch apply rgw SERVICE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch apply rgw foo
    Copy to Clipboard Toggle word wrap

Method 3

  • Use an arbitrary service name on a labeled set of hosts:

    Syntax

    ceph orch host label add HOST_NAME_1 LABEL_NAME
ceph orch host label add HOSTNAME_2 LABEL_NAME
ceph orch apply rgw SERVICE_NAME --placement="label:LABEL_NAME count-per-host:NUMBER_OF_DAEMONS" --port=8000
    Copy to Clipboard Toggle word wrap

    Note

    NUMBER_OF_DAEMONS controls the number of Ceph object gateways deployed on each host. To achieve the highest performance without incurring an additional cost, set this value to 2.

    Example

    [ceph: root@host01 /]# ceph orch host label add host01 rgw  # the 'rgw' label can be anything
[ceph: root@host01 /]# ceph orch host label add host02 rgw
[ceph: root@host01 /]# ceph orch apply rgw foo --placement="2 label:rgw" --port=8000
    Copy to Clipboard Toggle word wrap

Verification

  • List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls
    Copy to Clipboard Toggle word wrap

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps --daemon_type=DAEMON_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch ps --daemon_type=rgw
    Copy to Clipboard Toggle word wrap

You can deploy the Ceph Object Gateway using the service specification with either the default or the custom realms, zones, and zone groups.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the bootstrapped host.
  • Hosts are added to the cluster.
  • All manager, monitor, and OSD daemons are deployed.

Procedure

  1. As a root user, create a specification file:

    Example

    [root@host01 ~]# touch radosgw.yml
    Copy to Clipboard Toggle word wrap

  2. Configure S3 requests to wait for the duration defined in the rgw_exit_timeout_secs parameter for all outstanding requests to complete by setting rgw_graceful_stop to 'true' during Ceph Object gateway shutdown/restart.

    Syntax

    ceph config set client.rgw rgw_graceful_stop true

ceph config set client.rgw rgw_exit_timeout_secs 120
    Copy to Clipboard Toggle word wrap

    Note

    In containerized deployments, an additional extra_container_agrs configuration of --stop-timeout=120 (or the value of rgw_exit_timeout_secs configuration, if not default) is also necessary in order for it to work as expected with ceph orch stop/restart commands. 

    [root@host1 ~]$ cat rgw_spec.yaml
service_type: rgw
service_id: foo
placement:
  count_per_host: 1
  hosts:
    - rgw_node
spec:
  rgw_frontend_port: 8081
extra_container_args:
  - --stop-timeout=120
    Copy to Clipboard Toggle word wrap

  3. Edit the radosgw.yml file to include the following details for the default realm, zone, and zone group:

    Syntax

    service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - HOST_NAME_1
  - HOST_NAME_2
  count_per_host: NUMBER_OF_DAEMONS
spec:
  rgw_realm: REALM_NAME
  rgw_zone: ZONE_NAME
  rgw_frontend_port: FRONT_END_PORT
networks:
  -  NETWORK_CIDR # Ceph Object Gateway service binds to a specific network
    Copy to Clipboard Toggle word wrap

    Note

    NUMBER_OF_DAEMONS controls the number of Ceph Object Gateways deployed on each host. To achieve the highest performance without incurring an additional cost, set this value to 2.

    Example

    service_type: rgw
service_id: default
placement:
  hosts:
  - host01
  - host02
  - host03
  count_per_host: 2
spec:
  rgw_realm: default
  rgw_zone: default
  rgw_frontend_port: 1234
networks:
  - 192.169.142.0/24
    Copy to Clipboard Toggle word wrap

  4. Optional: For custom realm, zone, and zone group, create the resources and then create the radosgw.yml file:

    1. Create the custom realm, zone, and zone group:

      Example

      [root@host01 ~]# radosgw-admin realm create --rgw-realm=test_realm
[root@host01 ~]# radosgw-admin zonegroup create --rgw-zonegroup=test_zonegroup
[root@host01 ~]# radosgw-admin zone create --rgw-zonegroup=test_zonegroup --rgw-zone=test_zone
[root@host01 ~]# radosgw-admin period update --rgw-realm=test_realm --commit
      Copy to Clipboard Toggle word wrap

    2. Create the radosgw.yml file with the following details:

      Example

      service_type: rgw
service_id: test_realm.test_zone
placement:
  hosts:
  - host01
  - host02
  - host03
  count_per_host: 2
spec:
  rgw_realm: test_realm
  rgw_zone: test_zone
  rgw_frontend_port: 1234
networks:
  - 192.169.142.0/24
      Copy to Clipboard Toggle word wrap

  5. Mount the radosgw.yml file under a directory in the container:

    Example

    [root@host01 ~]# cephadm shell --mount radosgw.yml:/var/lib/ceph/radosgw/radosgw.yml
    Copy to Clipboard Toggle word wrap

    Note

    Every time you exit the shell, you have to mount the file in the container before deploying the daemon.

  6. Deploy the Ceph Object Gateway using the service specification:

    Syntax

    ceph orch apply -i FILE_NAME.yml
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch apply -i radosgw.yml
    Copy to Clipboard Toggle word wrap

Verification

  • List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls
    Copy to Clipboard Toggle word wrap

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps --daemon_type=DAEMON_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch ps --daemon_type=rgw
    Copy to Clipboard Toggle word wrap

Ceph Orchestrator supports multi-site configuration options for the Ceph Object Gateway.

You can configure each object gateway to work in an active-active zone configuration allowing writes to a non-primary zone. The multi-site configuration is stored within a container called a realm.

The realm stores zone groups, zones, and a time period. The rgw daemons handle the synchronization eliminating the need for a separate synchronization agent, thereby operating with an active-active configuration.

You can also deploy multi-site zones using the command line interface (CLI).

Note

The following configuration assumes at least two Red Hat Ceph Storage clusters are in geographically separate locations. However, the configuration also works on the same site.

Prerequisites

  • At least two running Red Hat Ceph Storage clusters.
  • At least two Ceph Object Gateway instances, one for each Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Nodes or containers are added to the storage cluster.
  • All Ceph Manager, Monitor and OSD daemons are deployed.

Procedure

  1. In the cephadm shell, configure the primary zone:

    1. Create a realm:

      Syntax

      radosgw-admin realm create --rgw-realm=REALM_NAME --default
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default
      Copy to Clipboard Toggle word wrap

      If the storage cluster has a single realm, then specify the --default flag.

    2. Create a primary zone group:

      Syntax

      radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --master --default
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default
      Copy to Clipboard Toggle word wrap

    3. Create a primary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=PRIMARY_ZONE_GROUP_NAME --rgw-zone=PRIMARY_ZONE_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ
      Copy to Clipboard Toggle word wrap

    4. Optional: Delete the default zone, zone group, and the associated pools.

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data. Also, removing the default zone group deletes the system user.

      To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup delete --rgw-zonegroup=default
[ceph: root@host01 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
      Copy to Clipboard Toggle word wrap

    5. Create a system user:

      Syntax

      radosgw-admin user create --uid=USER_NAME --display-name="USER_NAME" --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY --system
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin user create --uid=zone.user --display-name="Zone user" --system
      Copy to Clipboard Toggle word wrap

      Make a note of the access_key and secret_key.

    6. Add the access key and system key to the primary zone:

      Syntax

      radosgw-admin zone modify --rgw-zone=PRIMARY_ZONE_NAME --access-key=ACCESS_KEY --secret=SECRET_KEY
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=us-east-1 --access-key=NE48APYCAODEPLKBCZVQ--secret=u24GHQWRE3yxxNBnFBzjM4jn14mFIckQ4EKL6LoW
      Copy to Clipboard Toggle word wrap

    7. Commit the changes:

      Syntax

      radosgw-admin period update --commit
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# radosgw-admin period update --commit
      Copy to Clipboard Toggle word wrap

    8. Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host01 ~]#  systemctl list-units | grep ceph
      Copy to Clipboard Toggle word wrap

    9. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
systemctl enable ceph-FSID@DAEMON_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [root@host01 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service
[root@host01 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service
      Copy to Clipboard Toggle word wrap

  2. In the Cephadm shell, configure the secondary zone.

    1. Pull the primary realm configuration from the host:

      Syntax

      radosgw-admin realm pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host04 /]# radosgw-admin realm pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ
      Copy to Clipboard Toggle word wrap

    2. Pull the primary period configuration from the host:

      Syntax

      radosgw-admin period pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host04 /]# radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ
      Copy to Clipboard Toggle word wrap

    3. Configure a secondary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME \
             --rgw-zone=SECONDARY_ZONE_NAME --endpoints=http://RGW_SECONDARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 \
             --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY \
             --endpoints=http://FQDN:80 \
             [--read-only]
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --endpoints=http://rgw2:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ --endpoints=http://rgw.example.com:80
      Copy to Clipboard Toggle word wrap

    4. Optional: Delete the default zone.

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data.

      To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

      Example

      [ceph: root@host04 /]# radosgw-admin zone rm --rgw-zone=default
[ceph: root@host04 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
      Copy to Clipboard Toggle word wrap

    5. Update the Ceph configuration database:

      Syntax

      ceph config set SERVICE_NAME rgw_zone SECONDARY_ZONE_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host04 /]# ceph config set rgw rgw_zone us-east-2
      Copy to Clipboard Toggle word wrap

    6. Commit the changes:

      Syntax

      radosgw-admin period update --commit
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host04 /]# radosgw-admin period update --commit
      Copy to Clipboard Toggle word wrap

    7. Outside the Cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host04 ~]#  systemctl list-units | grep ceph
      Copy to Clipboard Toggle word wrap

    8. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
systemctl enable ceph-FSID@DAEMON_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [root@host04 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
[root@host04 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
      Copy to Clipboard Toggle word wrap

  3. Optional: Deploy multi-site Ceph Object Gateways using the placement specification:

    Syntax

    ceph orch apply rgw NAME --realm=REALM_NAME --zone=PRIMARY_ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host04 /]# ceph orch apply rgw east --realm=test_realm --zone=us-east-1 --placement="2 host01 host02"
    Copy to Clipboard Toggle word wrap

Verification

  • Check the synchronization status to verify the deployment:

    Example

    [ceph: root@host04 /]# radosgw-admin sync status
    Copy to Clipboard Toggle word wrap

You can remove the Ceph object gateway daemons using the ceph orch rm command.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • At least one Ceph object gateway daemon deployed on the hosts.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls
    Copy to Clipboard Toggle word wrap

  3. Remove the service:

    Syntax

    ceph orch rm SERVICE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch rm rgw.test_realm.test_zone_bb
    Copy to Clipboard Toggle word wrap

Verification

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch ps
    Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat