Chapter 2. Integrating Red Hat Ceph Storage

Procedure

1. Create pools for the Compute service (vms), the Block Storage service (volumes), and the Image service (images):

   $ for P in vms volumes images; do
     cephadm shell -- ceph osd pool create $P;
     cephadm shell -- ceph osd pool application enable $P rbd;
   done

2. Optional: Create the cephfs volume if the Shared File Systems service (manila) is enabled in the control plane. This automatically enables the CephFS Metadata service (MDS) and creates the necessary data and metadata pools on the Ceph cluster:

   $ cephadm shell -- ceph fs volume create cephfs

3. Optional: Deploy an NFS service on the Red Hat Ceph Storage cluster to use CephFS with NFS:

   $ cephadm shell -- ceph nfs cluster create cephfs \
   --ingress --virtual-ip=<vip> \
   --ingress-mode=haproxy-protocol

   • Replace <vip> with the IP address assigned to the NFS service. The NFS service should be isolated on a network that can be shared with all Red Hat OpenStack users. See NFS cluster and export management, for more information about customizing the NFS service.

     Important

     When you deploy an NFS service for the Shared File Systems service, do not select a custom port to expose NFS. Only the default NFS port of 2049 is supported. You must enable the Red Hat Ceph Storage ingress service and set the ingress-mode to haproxy-protocol. Otherwise, you cannot use IP-based access rules with the Shared File Systems service. For security in production environments, Red Hat does not recommend providing access to 0.0.0.0/0 on shares to mount them on client machines.

4. Create a cephx key for RHOSO to use to access pools:

   $ cephadm shell -- \
      ceph auth add client.openstack \
        mgr 'allow *' \
           mon 'allow r' \
           osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rwx pool=volumes, allow rwx pool=images'

   Important

   If the Shared File Systems service is enabled in the control plane, replace osd caps with the following:

   osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rwx pool=volumes, allow rwx pool=images, allow rwx pool=cephfs.cephfs.data'

5. Export the cephx key:

   $ cephadm shell -- ceph auth get client.openstack > /etc/ceph/ceph.client.openstack.keyring

6. Export the configuration file:

   $ cephadm shell -- ceph config generate-minimal-conf > /etc/ceph/ceph.conf

Procedure

1. Transfer the cephx key and configuration file created in the Creating Red Hat Ceph Storage pools procedure to a host that can create resources in the openstack namespace.

2. Base64 encode these files and store them in KEY and CONF environment variables:

   $ KEY=$(cat /etc/ceph/ceph.client.openstack.keyring | base64 -w 0)
   $ CONF=$(cat /etc/ceph/ceph.conf | base64 -w 0)

3. Create a YAML file to create the Secret resource.

4. Using the environment variables, add the Secret configuration to the YAML file:

   apiVersion: v1
   data:
     ceph.client.openstack.keyring: $KEY
     ceph.conf: $CONF
   kind: Secret
   metadata:
     name: ceph-conf-files
     namespace: openstack
   type: Opaque

5. Save the YAML file.

6. Create the Secret resource:

   $ oc create -f <secret_configuration_file>

   • Replace <secret_configuration_file> with the name of the YAML file you created.

1. Confirming the Red Hat Ceph Storage cluster and the associated services have the correct network configuration.
2. Configuring the control plane to use the Red Hat Ceph Storage secret.
3. Configuring the Image service (glance) to use the Red Hat Ceph Storage cluster.
4. Configuring the Block Storage service (cinder) to use the Red Hat Ceph Storage cluster.
5. Optional: Configuring the Shared File Systems service (manila) to use native CephFS or CephFS-NFS with the Red Hat Ceph Storage cluster.

Procedure

1. Check the storage interface defined in your NodeNetworkConfigurationPolicy (nncp) custom resource to confirm that it has the same network configuration as the public_network of the Red Hat Ceph Storage cluster. This is required to enable access to the Red Hat Ceph Storage cluster through the Storage network. The Storage network should have the same network configuration as the public_network of the Red Hat Ceph Storage cluster.

   Note

   It is not necessary for RHOSO to access the cluster_network of the Red Hat Ceph Storage cluster.

2. Check the networkAttachments for the default Image service instance in the OpenStackControlPlane CR to confirm that the default Image service is configured to access the Storage network:

   glance:
       enabled: true
       template:
         databaseInstance: openstack
         storageClass: ""
         storageRequest: 10G
         glanceAPIs:
           default
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             networkAttachments:
             - storage

3. Confirm the Block Storage service is configured to access the Storage network through MetalLB.
4. Optional: Confirm the Shared File Systems service is configured to access the Storage network through ManilaShare.
5. Confirm the Compute service (nova) is configured to access the Storage network.
6. Confirm the Red Hat Ceph Storage configuration file, /etc/ceph/ceph.conf, contains the IP addresses of the Red Hat Ceph Storage cluster monitors. These IP addresses must be within the Storage network IP address range.
7. Open your openstack_control_plane.yaml file to edit the OpenStackControlPlane CR.

8. Add the extraMounts parameter to define the services that require access to the Red Hat Ceph Storage secret.

   The following is an example of using the extraMounts parameter for this purpose. Only include ManilaShare in the propagation list if you are using the Shared File Systems service (manila):

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - CinderVolume
             - GlanceAPI
             - ManilaShare
             extraVolType: Ceph
             volumes:
             - name: ceph
               projected:
                 sources:
                 - secret:
                     name: <ceph-conf-files>
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

   • Replace <ceph-conf-files> with the name of your Secret CR created in Creating a Red Hat Ceph Storage secret.

9. Add the customServiceConfig parameter to the glance template to configure the Image service to use the Red Hat Ceph Storage cluster:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
     glance:
       enabled: true
       template:
         databaseInstance: openstack
         databaseUser: glance
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = default_backend:rbd
           enabled_import_methods=[web-download,glance-direct]
           [glance_store]
           default_backend = default_backend
           [default_backend]
           rbd_store_ceph_conf = /etc/ceph/ceph.conf
           store_description = "RBD backend"
           rbd_store_pool = images
           rbd_store_user = openstack
       glanceAPIs:
         default:
           preserveJobs: false
           replicas: 1
       secret: osp-secret
       storageClass: ""
       storageRequest: 10G
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - Glance
             extraVolType: Ceph
             volumes:
               - name: ceph
                 projected:
                   sources:
                   - secret:
                       name: ceph-conf-files
             mounts:
               - name: ceph
                 mountPath: "/etc/ceph"
                 readOnly: true

10. Add the customServiceConfig parameter to the cinder template to configure the Block Storage service to use the Red Hat Ceph Storage cluster:

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
        ...
      cinder:
        template:
          cinderVolumes:
            ceph:
              customServiceConfig: |
                [DEFAULT]
                enabled_backends=ceph
                [ceph]
                volume_backend_name=ceph
                volume_driver=cinder.volume.drivers.rbd.RBDDriver
                rbd_ceph_conf=/etc/ceph/ceph.conf
                rbd_user=openstack
                rbd_pool=volumes
                rbd_flatten_volume_from_snapshot=False
                rbd_secret_uuid=$FSID 1

    1

    Replace with the actual FSID. The FSID itself does not need to be considered secret. For more information, see Obtaining the Red Hat Ceph Storage FSID.

11. Optional: Add the customServiceConfig parameter to the manila template to configure the Shared File Systems service to use native CephFS or CephFS-NFS with the Red Hat Ceph Storage cluster. For more information, see Configuring the Shared File Systems service (manila):

    The following example exposes native CephFS:

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
        ...
        manila:
            template:
     	manilaAPI:
                 customServiceConfig: |
                   [DEFAULT]
                   enabled_share_protocols=cephfs
               manilaShares:
                 share1:
                    customServiceConfig: |
                        [DEFAULT]
                        enabled_share_backends=cephfs
                        [cephfs]
                        driver_handles_share_servers=False
                        share_backend_name=cephfs
                        share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                        cephfs_conf_path=/etc/ceph/ceph.conf
                        cephfs_auth_id=openstack
                        cephfs_cluster_name=ceph
                        cephfs_volume_mode=0755
                        cephfs_protocol_helper_type=CEPHFS

    The following example exposes CephFS with NFS:

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
        ...
        manila:
            template:
               manilaAPI:
                 customServiceConfig: |
                   [DEFAULT]
                   enabled_share_protocols=nfs
               manilaShares:
                 share1:
                    customServiceConfig: |
                        [DEFAULT]
                        enabled_share_backends=cephfsnfs
                        [cephfsnfs]
                        driver_handles_share_servers=False
                        share_backend_name=cephfsnfs
                        share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                        cephfs_conf_path=/etc/ceph/ceph.conf
                        cephfs_auth_id=openstack
                        cephfs_cluster_name=ceph
                        cephfs_volume_mode=0755
                        cephfs_protocol_helper_type=NFS
                        cephfs_nfs_cluster_id=cephfs

12. Apply the updates to the OpenStackControlPlane CR:

    $ oc apply -f openstack_control_plane.yaml

Procedure

1. Create a ConfigMap with additional content for the Compute service (nova) configuration file /etc/nova/nova.conf.d/ inside the nova_compute container. This additional content directs the Compute service to use Red Hat Ceph Storage RBD.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ceph-nova
   data:
     03-ceph-nova.conf: | 1
       [libvirt]
       images_type=rbd
       images_rbd_pool=vms
       images_rbd_ceph_conf=/etc/ceph/ceph.conf
       images_rbd_glance_store_name=default_backend
       images_rbd_glance_copy_poll_interval=15
       images_rbd_glance_copy_timeout=600
       rbd_user=openstack
       rbd_secret_uuid=$FSID 2

   1

   This file name must follow the naming convention of ##-<name>-nova.conf. Files are evaluated by the Compute service alphabetically. A filename that starts with 01 will be evaluated by the Compute service before a filename that starts with 02.

   2

   The $FSID value should contain the actual FSID as described in the Obtaining the Ceph FSID section. The FSID itself does not need to be considered secret.

2. Create a custom version of the default nova operator to use the new ConfigMap, which in this case is called ceph-nova.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ceph 1
   spec:
     label: dataplane-deployment-nova-custom-ceph
     configMaps:
       - ceph-nova
     secrets:
       - nova-cell1-compute-config
     playbook: osp.edpm.nova

   1

   The custom service is named nova-custom-ceph. It cannot be named nova because nova is an unchangeable default service. Any custom service that has the same name as a default service name will be overwritten during reconciliation.

3. Apply the ConfigMap and custom service changes:

   $ oc create -f ceph-nova.yaml

4. Update the OpenStackDataPlaneNodeSet services list to replace the nova service with the new custom service (in this case called nova-custom-ceph), add the ceph-client service, and use the extraMounts parameter to define access to the Ceph Storage secret.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   spec:
     ...
     roles:
       edpm-compute:
         ...
         services:
           - configure-network
           - validate-network
           - install-os
           - configure-os
           - run-os
           - ceph-client
           - ovn
           - libvirt
           - nova-custom-ceph
           - telemetry
   
     nodeTemplate:
       extraMounts:
       - extraVolType: Ceph
         volumes:
         - name: ceph
           secret:
             secretName: ceph-conf-files
         mounts:
         - name: ceph
           mountPath: "/etc/ceph"
           readOnly: true

   Note

   The ceph-client service must be added before the libvirt and nova-custom-ceph services. The ceph-client service configures EDPM nodes as clients of a Red Hat Ceph Storage server by distributing the Red Hat Ceph Storage client files.

5. Save the changes to the services list.

6. Create an OpenStackDataPlaneDeployment CR:

   $ oc create -f <dataplanedeployment_cr_file>

   • Replace <dataplanedeployment_cr_file> with the name of your file.

     Note

     An example of an OpenStackDataPlaneDeployment CR file is available here: link:https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/config/samples/dataplane_v1beta1_openstackdataplanedeployment.yaml.

1. Configure the RGW to verify users and their roles in the Identity service (keystone) to authenticate with the external RGW service.
2. Deploy and configure a RGW service to handle object storage requests.