Chapter 7. Migrating the Red Hat Ceph Storage cluster

Procedure

1. Add the RGW label to the Red Hat Ceph Storage nodes that you want to migrate your RGW back ends to:

   $ sudo cephadm shell -- ceph orch host label add cephstorage-0 rgw;
   $ sudo cephadm shell -- ceph orch host label add cephstorage-1 rgw;
   $ sudo cephadm shell -- ceph orch host label add cephstorage-2 rgw;
   
   Added label rgw to host cephstorage-0
   Added label rgw to host cephstorage-1
   Added label rgw to host cephstorage-2
   
   $ sudo cephadm shell -- ceph orch host ls
   
   HOST       	ADDR       	LABELS      	STATUS
   cephstorage-0  192.168.24.54  osd rgw
   cephstorage-1  192.168.24.44  osd rgw
   cephstorage-2  192.168.24.30  osd rgw
   controller-0   192.168.24.45  _admin mon mgr
   controller-1   192.168.24.11  _admin mon mgr
   controller-2   192.168.24.38  _admin mon mgr
   
   6 hosts in cluster

2. Locate the RGW spec and dump in the spec directory:

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ mkdir -p ${SPEC_DIR}
   $ sudo cephadm shell -- ceph orch ls --export rgw > ${SPEC_DIR}/rgw
   $ cat ${SPEC_DIR}/rgw

   networks:
   - 172.17.3.0/24
   placement:
     hosts:
     - controller-0
     - controller-1
     - controller-2
   service_id: rgw
   service_name: rgw.rgw
   service_type: rgw
   spec:
     rgw_frontend_port: 8080
     rgw_realm: default
     rgw_zone: default

   This example assumes that 172.17.3.0/24 is the storage network.

3. In the placement section, ensure that the label and rgw_frontend_port values are set:

   ---
   networks:
   - 172.17.3.0/24
   placement:
     label: rgw
   service_id: rgw
   service_name: rgw.rgw
   service_type: rgw
   spec:
     rgw_frontend_port: 8090
     rgw_realm: default
     rgw_zone: default
     rgw_frontend_ssl_certificate: ...
     ssl: true

   • networks defines the storage network where the RGW back ends are deployed.
   • placement.label: rgw replaces the Controller nodes with the rgw label.
   • spec.rgw_frontend_port specifies the value as 8090 to avoid conflicts with the Ceph ingress daemon.
   • spec.rgw_frontend_ssl_certificate defines the SSL certificate and key concatenation if TLS is enabled as described in Configuring RGW with TLS for an external Red Hat Ceph Storage cluster in Configuring persistent storage.

4. Apply the new RGW spec by using the orchestrator CLI:

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ sudo cephadm shell -m ${SPEC_DIR}/rgw -- ceph orch apply -i /mnt/rgw

   This command triggers the redeploy, for example:

   ...
   osd.9                     	cephstorage-2
   rgw.rgw.cephstorage-0.wsjlgx  cephstorage-0  172.17.3.23:8090   starting
   rgw.rgw.cephstorage-1.qynkan  cephstorage-1  172.17.3.26:8090   starting
   rgw.rgw.cephstorage-2.krycit  cephstorage-2  172.17.3.81:8090   starting
   rgw.rgw.controller-1.eyvrzw   controller-1   172.17.3.146:8080  running (5h)
   rgw.rgw.controller-2.navbxa   controller-2   172.17.3.66:8080   running (5h)
   
   ...
   osd.9                     	cephstorage-2
   rgw.rgw.cephstorage-0.wsjlgx  cephstorage-0  172.17.3.23:8090  running (19s)
   rgw.rgw.cephstorage-1.qynkan  cephstorage-1  172.17.3.26:8090  running (16s)
   rgw.rgw.cephstorage-2.krycit  cephstorage-2  172.17.3.81:8090  running (13s)

5. Ensure that the new RGW back ends are reachable on the new ports, so you can enable an ingress daemon on port 8080 later. Log in to each Red Hat Ceph Storage node that includes RGW and add the iptables rule to allow connections to both 8080 and 8090 ports in the Red Hat Ceph Storage nodes:

   $ iptables -I INPUT -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW -m comment --comment "ceph rgw ingress" -j ACCEPT
   $ iptables -I INPUT -p tcp -m tcp --dport 8090 -m conntrack --ctstate NEW -m comment --comment "ceph rgw backends" -j ACCEPT
   $ sudo iptables-save
   $ sudo systemctl restart iptables

6. If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content:

   # 100 ceph_rgw {'dport': ['8080','8090']}
   add rule inet filter TRIPLEO_INPUT tcp dport { 8080,8090 } ct state new counter accept comment "100 ceph_rgw"

7. Save the file.

8. Restart the nftables service:

   $ sudo systemctl restart nftables

9. Verify that the rules are applied:

   $ sudo nft list ruleset | grep ceph_rgw

10. From a Controller node, such as controller-0, try to reach the RGW back ends:

    $ curl http://cephstorage-0.storage:8090;

    You should observe the following output:

    <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

    Repeat the verification for each node where a RGW daemon is deployed.

11. If you migrated RGW back ends to the Red Hat Ceph Storage nodes, there is no internalAPI network, except in the case of HCI nodes. You must reconfigure the RGW keystone endpoint to point to the external network that you propagated:

    [ceph: root@controller-0 /]# ceph config dump | grep keystone
    global   basic rgw_keystone_url  http://172.16.1.111:5000
    
    [ceph: root@controller-0 /]# ceph config set global rgw_keystone_url http://<keystone_endpoint>:5000

    • Replace <keystone_endpoint> with the Identity service (keystone) internal endpoint of the service that is deployed in the OpenStackControlPlane CR when you adopt the Identity service. For more information, see Adopting the Identity service.

1. Remove the existing ceph_rgw configuration.
2. Clean up the configuration created by director.
3. Redeploy the Object Storage service (swift).

Procedure

1. Log in to each Controller node and remove the following configuration from the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file:

   listen ceph_rgw
     bind 10.0.0.103:8080 transparent
     mode http
     balance leastconn
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
     http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
     http-request set-header X-Forwarded-Port %[dst_port]
     option httpchk GET /swift/healthcheck
     option httplog
     option forwardfor
      server controller-0.storage.redhat.local 172.17.3.73:8080 check fall 5 inter 2000 rise 2
     server controller-1.storage.redhat.local 172.17.3.146:8080 check fall 5 inter 2000 rise 2
     server controller-2.storage.redhat.local 172.17.3.156:8080 check fall 5 inter 2000 rise 2

2. Restart haproxy-bundle and confirm that it is started:

   [root@controller-0 ~]# sudo pcs resource restart haproxy-bundle
   haproxy-bundle successfully restarted
   
   
   [root@controller-0 ~]# sudo pcs status | grep haproxy
   
     * Container bundle set: haproxy-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-haproxy:pcmklatest]:
       * haproxy-bundle-podman-0   (ocf:heartbeat:podman):  Started controller-0
       * haproxy-bundle-podman-1   (ocf:heartbeat:podman):  Started controller-1
       * haproxy-bundle-podman-2   (ocf:heartbeat:podman):  Started controller-2

3. Confirm that no process is connected to port 8080:

   [root@controller-0 ~]# ss -antop | grep 8080
   [root@controller-0 ~]#

   You can expect the Object Storage service (swift) CLI to fail to establish the connection:

   (overcloud) [root@cephstorage-0 ~]# swift list
   
   HTTPConnectionPool(host='10.0.0.103', port=8080): Max retries exceeded with url: /swift/v1/AUTH_852f24425bb54fa896476af48cbe35d3?format=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc41beb0430>: Failed to establish a new connection: [Errno 111] Connection refused'))

4. Set the required images for both HAProxy and Keepalived:

   [ceph: root@controller-0 /]# ceph config set mgr mgr/cephadm/container_image_haproxy registry.redhat.io/rhceph/rhceph-haproxy-rhel9:latest
   [ceph: root@controller-0 /]# ceph config set mgr mgr/cephadm/container_image_keepalived registry.redhat.io/rhceph/keepalived-rhel9:latest

5. Create a file called rgw_ingress in controller-0:

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ vim ${SPEC_DIR}/rgw_ingress

6. Paste the following content into the rgw_ingress file:

   ---
   service_type: ingress
   service_id: rgw.rgw
   placement:
     label: rgw
   spec:
     backend_service: rgw.rgw
     virtual_ip: 10.0.0.89/24
     frontend_port: 8080
     monitor_port: 8898
     virtual_interface_networks:
       - <external_network>
     ssl_cert: ...

   • Replace <external_network> with your external network, for example, 10.0.0.0/24. For more information, see Completing prerequisites for migrating Red Hat Ceph Storage RGW.
   • If TLS is enabled, add the SSL certificate and key concatenation as described in Configuring RGW with TLS for an external Red Hat Ceph Storage cluster in Configuring persistent storage.

7. Apply the rgw_ingress spec by using the Ceph orchestrator CLI:

   $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
   $ cephadm shell -m ${SPEC_DIR}/rgw_ingress -- ceph orch apply -i /mnt/rgw_ingress

8. Wait until the ingress is deployed and query the resulting endpoint:

   $ sudo cephadm shell -- ceph orch ls
   
   NAME                 	PORTS            	RUNNING  REFRESHED  AGE  PLACEMENT
   crash                                         	6/6  6m ago 	3d   *
   ingress.rgw.rgw      	10.0.0.89:8080,8898  	6/6  37s ago	60s  label:rgw
   mds.mds                   3/3  6m ago 	3d   controller-0;controller-1;controller-2
   mgr                       3/3  6m ago 	3d   controller-0;controller-1;controller-2
   mon                       3/3  6m ago 	3d   controller-0;controller-1;controller-2
   osd.default_drive_group   15  37s ago	3d   cephstorage-0;cephstorage-1;cephstorage-2
   rgw.rgw   ?:8090          3/3  37s ago	4m   label:rgw

   $ curl 10.0.0.89:8080
   
   ---
   <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[ceph: root@controller-0 /]#
   —

Procedure

1. List the current swift endpoints and service:

    $ oc rsh openstackclient openstack endpoint list | grep "swift.*object"

    $ oc rsh openstackclient openstack service list | grep "swift.*object"

2. If the service and endpoints do not exist, create the missing swift resources:

   $ oc rsh openstackclient openstack service create --name swift --description 'OpenStack Object Storage' object-store
   $ oc rsh openstackclient openstack role add --user swift --project service member
   $ oc rsh openstackclient openstack role add --user swift --project service admin
   > for i in public internal; do
   >     oc rsh openstackclient endpoint create --region regionOne object-store $i http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s
   > done
   $ oc rsh openstackclient openstack role add --project admin --user admin swiftoperator

   • Replace <RGW_VIP> with the Ceph RGW ingress VIP.

3. If the endpoints exist, update the endpoints to point to the right RGW ingress VIP:

   $ oc rsh openstackclient openstack endpoint set --url http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s <swift_public_endpoint_uuid>
   $ oc rsh openstackclient openstack endpoint set --url http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s <swift_internal_endpoint_uuid>
   $ oc rsh openstackclient openstack endpoint list | grep object
   | 0d682ad71b564cf386f974f90f80de0d | regionOne | swift        | object-store | True    | public    | http://172.18.0.100:8080/swift/v1/AUTH_%(tenant_id)s    |
   | b311349c305346f39d005feefe464fb1 | regionOne | swift        | object-store | True    | internal  | http://172.18.0.100:8080/swift/v1/AUTH_%(tenant_id)s    |

   • Replace <swift_public_endpoint_uuid> with the UUID of the swift public endpoint.
   • Replace <swift_internal_endpoint_uuid> with the UUID of the swift internal endpoint.

4. Test the migrated service:

   $ oc rsh openstackclient openstack container list --debug
   
   ...
   ...
   ...
   REQ: curl -g -i -X GET http://keystone-public-openstack.apps.ocp.openstack.lab -H "Accept: application/json" -H "User-Agent: openstacksdk/1.0.2 keystoneauth1/5.1.3 python-requests/2.25.1 CPython/3.9.23"
   Starting new HTTP connection (1): keystone-public-openstack.apps.ocp.openstack.lab:80
   http://keystone-public-openstack.apps.ocp.openstack.lab:80 "GET / HTTP/1.1" 300 298
   RESP: [300] content-length: 298 content-type: application/json date: Mon, 14 Jul 2025 17:41:29 GMT location: http://keystone-public-openstack.apps.ocp.openstack.lab/v3/ server: Apache set-cookie: b5697f82cf3c19ece8be533395142512=d5c6a9ee2
   267c4b63e9f656ad7565270; path=/; HttpOnly vary: X-Auth-Token x-openstack-request-id: req-452e42c5-e60f-440f-a6e8-fe1b9ea89055
   RESP BODY: {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://keystone-public-openstack.apps.ocp.openstack.lab/v3/"}], "media-types": [{"base": "applic
   ation/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
   GET call to http://keystone-public-openstack.apps.ocp.openstack.lab/ used request id req-452e42c5-e60f-440f-a6e8-fe1b9ea89055
   
   ...
   
   REQ: curl -g -i -X GET "http://172.18.0.100:8080/swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json" -H "User-Agent: openstacksdk/1.0.2 keystoneauth1/5.1.3 python-requests/2.25.1 CPython/3.9.23" -H "X-Auth-Token: {SHA256}ec5deca0be37bd8bfe659f132b9cdf396b8f409db5dc16972d50cbf3f28474d4"
   Starting new HTTP connection (1): 172.18.0.100:8080
   http://172.18.0.100:8080 "GET /swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json HTTP/1.1" 200 2
   RESP: [200] accept-ranges: bytes content-length: 2 content-type: application/json; charset=utf-8 date: Mon, 14 Jul 2025 17:41:31 GMT x-account-bytes-used: 0 x-account-bytes-used-actual: 0 x-account-container-count: 0 x-account-object-count: 0 x-account-storage-policy-default-placement-bytes-used: 0 x-account-storage-policy-default-placement-bytes-used-actual: 0 x-account-storage-policy-default-placement-container-count: 0 x-account-storage-policy-default-placement-object-count: 0 x-openstack-request-id: tx000001e95361131ccf694-006875414a-7753-default x-timestamp: 1752514891.25991 x-trans-id: tx000001e95361131ccf694-006875414a-7753-default
   RESP BODY: []
   GET call to http://172.18.0.100:8080/swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json used request id tx000001e95361131ccf694-006875414a-7753-default
   
   clean_up ListContainer:
   END return value: 0

Procedure

1. SSH into the target node and enable the firewall rules that are required to reach a Ceph Manager service:

   dports="6800:7300"
   ssh heat-admin@<target_node> sudo iptables -I INPUT \
       -p tcp --match multiport --dports $dports -j ACCEPT;

   • Replace <target_node> with the hostname of the hosts that are listed in the Red Hat Ceph Storage environment. Run ceph orch host ls to see the list of the hosts.

     Repeat this step for each target node.

2. Check that the rules are properly applied to the target node and persist them:

   $ sudo iptables-save
   $ sudo systemctl restart iptables

   Note

   The default dashboard port for ceph-mgr in a greenfield deployment is 8443. With director-deployed Red Hat Ceph Storage, the default port is 8444 because the service ran on the Controller node, and it was necessary to use this port to avoid a conflict. For adoption, update the dashboard port to 8443 in the ceph-mgr configuration and firewall rules.

3. Log in to controller-0 and update the dashboard port in the ceph-mgr configuration to 8443:

   $ sudo cephadm shell
   $ ceph config set mgr mgr/dashboard/server_port 8443
   $ ceph config set mgr mgr/dashboard/ssl_server_port 8443
   $ ceph mgr module disable dashboard
   $ ceph mgr module enable dashboard

4. If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content:

   # 113 ceph_mgr {'dport': ['6800-7300', 8443]}
   add rule inet filter TRIPLEO_INPUT tcp dport { 6800-7300,8443 } ct state new counter accept comment "113 ceph_mgr"

5. Save the file.

6. Restart the nftables service:

   $ sudo systemctl restart nftables

7. Verify that the rules are applied:

   $ sudo nft list ruleset | grep ceph_mgr

8. Prepare the target node to host the new Ceph Manager daemon, and add the mgr label to the target node:

   $ sudo cephadm shell -- ceph orch host label add <target_node> mgr

9. Repeat steps 1-7 for each target node that hosts a Ceph Manager daemon.

10. Get the Ceph Manager spec:

    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
    $ mkdir -p ${SPEC_DIR}
    $ sudo cephadm shell -- ceph orch ls --export mgr > ${SPEC_DIR}/mgr

11. Edit the retrieved spec and add the label: mgr section to the placement section:

    service_type: mgr
    service_id: mgr
    placement:
      label: mgr

12. Save the spec.

13. Apply the spec with cephadm by using the Ceph Orchestrator:

    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
    $ sudo cephadm shell -m ${SPEC_DIR}/mgr -- ceph orch apply -i /mnt/mgr

Verification

1. Verify that the new Ceph Manager daemons are created in the target nodes:

   $ sudo cephadm shell -- ceph orch ps | grep -i mgr
   $ sudo cephadm shell -- ceph -s

   The Ceph Manager daemon count should match the number of hosts where the mgr label is added.

   Note

   The migration does not shrink the Ceph Manager daemons. The count grows by the number of target nodes, and migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes decommissions the stand-by Ceph Manager instances. For more information, see Migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes.

1. Configure the target nodes for Ceph Monitor migration.
2. Drain the source node
3. Migrate your Ceph Monitor IP addresses to the target nodes
4. Redeploy the Ceph Monitor on the target node
5. Verify that the Red Hat Ceph Storage cluster is healthy