Integrating an Overcloud with an Existing Red Hat Ceph Storage Cluster

Procedure

1. Log in to the external Ceph admin node.

2. Open an interactive shell to access Ceph commands:

   [user@ceph ~]$ sudo cephadm shell

3. Create the following RADOS Block Device (RBD) pools in your Ceph Storage cluster, relevant to your environment:

   • Storage for OpenStack Block Storage (cinder):

     $ ceph osd pool create volumes <pgnum>

   • Storage for OpenStack Image Storage (glance):

     $ ceph osd pool create images <pgnum>

   • Storage for instances:

     $ ceph osd pool create vms <pgnum>

   • Storage for OpenStack Block Storage Backup (cinder-backup):

     $ ceph osd pool create backups <pgnum>

4. If your overcloud deploys the Shared File Systems service (manila) with Red Hat Ceph 5 (Ceph package 16) or later, you do not need to create data and metadata pools for CephFS. You can create a filesystem volume. For more information, see Management of MDS service using the Ceph Orchestrator in the Red Hat Ceph Storage Operations Guide.

5. Create a client.openstack user in your Ceph Storage cluster with the following capabilities:

   • cap_mgr: allow *
   • cap_mon: profile rbd

   • cap_osd: profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups

     $ ceph auth add client.openstack mgr 'allow *' mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups'

6. Note the Ceph client key created for the client.openstack user:

   $ ceph auth list
   ...
   [client.openstack]
   	key = <AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==>
   	caps mgr = "allow *"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups"
   ...

   • The key value in the example, AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==, is your Ceph client key.

7. If your overcloud deploys the Shared File Systems service with CephFS, create the client.manila user in your Ceph Storage cluster. The capabilities required for the client.manila user depend on whether your deployment exposes CephFS shares through the native CephFS protocol or the NFS protocol.

   • If you expose CephFS shares through the native CephFS protocol, the following capabilities are required:

     • cap_mgr: allow rw

     • cap_mon: allow r

       $ ceph auth add client.manila mgr 'allow rw' mon 'allow r'

   • If you expose CephFS shares through the NFS protocol, the following capabilities are required:

     • cap_mgr: allow rw
     • cap_mon: allow r

     • cap_osd: allow rw pool=manila_data

       The specified pool name must be the value set for the ManilaCephFSDataPoolName parameter, which defaults to manila_data.

       $ ceph auth add client.manila  mgr 'allow rw' mon 'allow r' osd 'allow rw pool=manila_data'

8. Note the manila client name and the key value to use in overcloud deployment templates:

   $ ceph auth get-key client.manila
        <AQDQ991cAAAAABAA0aXFrTnjH9aO39P0iVvYyg==>

9. Note the file system ID of your Ceph Storage cluster. This value is specified in the fsid field, under the [global] section of the configuration file for your cluster:

   [global]
   fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
   ...

Procedure

1. Create a custom environment file:

   /home/stack/templates/ceph-config.yaml

2. Add a parameter_defaults: section to the file:

   parameter_defaults:

3. Use parameter_defaults to set all of the parameters that you want to override in /usr/share/openstack-tripleo-heat-templates/environments/external-ceph.yaml. You must set the following parameters at a minimum:

   • CephClientKey: The Ceph client key for the client.openstack user in your Ceph Storage cluster. This is the value of key that you retrieved in Configuring the existing Ceph Storage cluster. For example, AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==.
   • CephClusterFSID: The file system ID of your Ceph Storage cluster. This is the value of fsid in your Ceph Storage cluster configuration file, which you retrieved in Configuring the existing Ceph Storage cluster. For example, 4b5c8c0a-ff60-454b-a1b4-9747aa737d19.

   • CephExternalMonHost: A comma-delimited list of the IPs of all MON hosts in your Ceph Storage cluster, for example, 172.16.1.7, 172.16.1.8.

     For example:

     parameter_defaults:
       CephClientKey: <AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==>
       CephClusterFSID: <4b5c8c0a-ff60-454b-a1b4-9747aa737d19>
       CephExternalMonHost: <172.16.1.7, 172.16.1.8, 172.16.1.9>

4. Optional: You can override the Red Hat OpenStack Platform (RHOSP) client username and the following default pool names to match your Ceph Storage cluster:

   • CephClientUserName: <openstack>
   • NovaRbdPoolName: <vms>
   • CinderRbdPoolName: <volumes>
   • GlanceRbdPoolName: <images>
   • CinderBackupRbdPoolName: <backups>

5. Optional: If you are deploying the Shared File Systems service with CephFS, you can override the following default data and metadata pool names:

     ManilaCephFSDataPoolName: <manila_data>
     ManilaCephFSMetadataPoolName: <manila_metadata>

   Note

   Ensure that these names match the names of the pools you created.

6. Set the client key that you created for the Shared File Systems service. You can override the default Ceph client username for that key:

     ManilaCephFSCephFSAuthId: <manila>
     CephManilaClientKey: <AQDQ991cAAAAABAA0aXFrTnjH9aO39P0iVvYyg==>

   Note

   The default client username ManilaCephFSCephFSAuthId is manila, unless you override it. CephManilaClientKey is always required.

Procedure

1. Create an image by using the Image service (glance). For more information about how to create an image, see Importing an image in Creating and Managing Images.
2. Record the image ID for later use.
3. Create a Compute (nova) instance. For more information about how to create an instance, see Creating an instance in the Creating and Managing Instances guide.
4. Record the instance ID for later use.
5. Create a Block Storage (cinder) volume. FFor more information about how to create a Block Storage volume, see Creating Block Storage volumes in the Storage Guide.
6. Record the volume ID for later use.
7. Create a file share by using the Shared File Systems service (manila).
8. List the export path of the share and record the UUID in the suffix for later use.

Procedure

1. Log in to the undercloud as the stack user and source the stackrc credentials file:

   $ source ~/stackrc

2. List the available servers to retrieve the IP addresses of nodes on the system:

   $ metalsmith list
   
   +---------------+----------------+---------------+
   | ID | Name | Status | Networks | Image | Flavor |
   +---------------+----------------+---------------+
   | d5a621bd-d109-41ae-a381-a42414397802 | compute-0 | ACTIVE | ctlplane=192.168.24.31 | overcloud-full | compute |
   | 496ab196-d6cb-447d-a118-5bafc5166cf2 | controller-0 | ACTIVE | ctlplane=192.168.24.37 | overcloud-full | controller |
   | c01e730d-62f2-426a-a964-b31448f250b3 | controller-2 | ACTIVE | ctlplane=192.168.24.55 | overcloud-full | controller |
   | 36df59b3-66f3-452e-9aec-b7e7f7c54b86 | controller-1 | ACTIVE | ctlplane=192.168.24.39 | overcloud-full | controller |
   | f8f00497-246d-4e40-8a6a-b5a60fa66483 | compute-1 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | compute |

3. Use SSH to log in to any Compute node:

   $ ssh tripleo-admin@192.168.24.31

4. Confirm that the files ceph.conf and /ceph.client.openstack.keyring exist in the CephConfigPath provided by director. This path is /var/lib/tripleo-config/ceph by default but an override might exist.

   [tripleo-admin@compute-0 ~]$ sudo ls -l /var/lib/tripleo-config/ceph/ceph.conf
   
   -rw-r--r--. 1 root root 1170 Sep 29 23:25 /var/lib/tripleo-config/ceph/ceph.conf
   
   [tripleo-admin@compute-0 ~]$ sudo ls -l /var/lib/tripleo-config/ceph/ceph.client.openstack.keyring
   
   -rw-------. 1 ceph ceph 253 Sep 29 23:25 /var/lib/tripleo-config/ceph/ceph.client.openstack.keyring

5. Enter the following command to force the nova_compute container to use the rbd command to list the contents of the appropriate pool.

   $ sudo podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls vms

   The pool name must match the pool names of the images, VMs, volumes, and shares that you created when you configured the Ceph Storage cluster. The IDs of the image, Compute instance, volume, and share must match the IDs that you recorded in Gathering IDs.

   Note

   The example command is prefixed with podman exec nova_compute because /usr/bin/rbd, which is provided by the ceph-common package, is not installed on overcloud nodes by default. However, it is available in the nova_compute container. The command lists block device images. For more information about listing block device images, see Listing the block device images in the Red Hat Ceph Storage Block Device Guide.

   The following examples show how to confirm whether an ID for each service is present for each pool by using the IDs from Gathering IDs.

   $ sudo podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls images | grep 4485d4c0-24c3-42ec-a158-4d3950fa020b

   $ sudo podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls vms | grep 64bcb731-e7a4-4dd5-a807-ee26c669482f

   $ sudo podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls volumes | grep aeac15e8-b67f-454f-9486-46b3d75daff4

6. To verify the existence of the Shared File Systems service share, you must log into a Controller node:

   $ sudo podman exec openstack-manila-share-podman-0 ceph -n client.manila fs subvolume ls cephfs | grep ec99db3c-0077-40b7-b09e-8a110e3f73c1

Procedure

1. To shorten the amount of typing you must do in this procedure, log in to a Compute node and create an alias for the rbd command:

   $ alias rbd="podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack"

2. Confirm that you can write test data to the pool as a new object:

   $ rbd create --size 1024 vms/foo

3. Confirm that you can see the test data:

   $ rbd ls vms | grep foo

4. Delete the test data:

   $ rbd rm vms/foo