Red Hat Summit Red Hat SummitSupportoConsoleSviluppatoriInizia una provaContatti

Questo contenuto non è disponibile nella lingua selezionata.

Chapter 2. CephFS through NFS Installation

2.1. CephFS with NFS-Ganesha deployment
Copia collegamento

A typical Ceph file system (CephFS) through NFS installation in a Red Hat OpenStack Platform (RHOSP) environment includes the following configurations:

  • OpenStack Controller nodes running containerized Ceph metadata server (MDS), Ceph monitor (MON), manila, and NFS-Ganesha services. Some of these services can coexist on the same node or can have one or more dedicated nodes.
  • Ceph storage cluster with containerized object storage daemons (OSDs) running on Ceph storage nodes.
  • An isolated StorageNFS network that provides access from projects to the NFS-Ganesha services for NFS share provisioning.
Important

The Shared File Systems service (manila) with CephFS through NFS fully supports serving shares to Red Hat OpenShift Container Platform through Manila CSI. This solution is not intended for large scale deployments. For important recommendations, see https://access.redhat.com/articles/6667651.

The Shared File Systems service (manila) provides APIs that allow the projects to request file system shares, which are fulfilled by driver modules. The driver for Red Hat CephFS, manila.share.drivers.cephfs.driver.CephFSDriver, means that you can use the Shared File Systems service as a CephFS back end. RHOSP director configures the driver to deploy the NFS-Ganesha gateway so that the CephFS shares are presented through the NFS 4.1 protocol.

When you use RHOSP director to deploy the Shared File Systems service with a CephFS back end on the overcloud, director automatically creates the required storage network that is defined in the heat template. For more information about network planning, see Overcloud networks in the Director Installation and Usage guide.

Although you can manually configure the Shared File Systems service by editing its node /etc/manila/manila.conf file, RHOSP director can override any settings in future overcloud updates. The recommended method for configuring a Shared File Systems back end is through director.

Note

Adding CephFS through NFS to an externally deployed Ceph cluster, which was not configured by Red Hat OpenStack Platform (RHOSP) director, is supported. Currently, only one CephFS back end can be defined in director. For more information, see Integrate with an existing Ceph Storage cluster in the Integrating an Overcloud with an Existing Red Hat Ceph Cluster guide.

2.1.1. Requirements
Copia collegamento

CephFS through NFS has been fully supported since Red Hat OpenStack Platform version (RHOSP) 13. The RHOSP Shared File Systems service with CephFS through NFS for RHOSP 16.0 and later is supported for use with Red Hat Ceph Storage version 4.1 or later. For more information about how to determine the version of Ceph Storage installed on your system, see Red Hat Ceph Storage releases and corresponding Ceph package versions.

Prerequisites

  • You install the Shared File Systems service on Controller nodes, as is the default behavior.
  • You install the NFS-Ganesha gateway service on Pacemaker cluster of the Controller node.
  • You configure only a single instance of a CephFS back end to use the Shared File Systems service. You can use other non-CephFS back ends with the single CephFS back end.
  • You use RHOSP director to create an extra network (StorageNFS) for the storage traffic.

2.1.2. File shares
Copia collegamento

File shares are handled slightly different between the OpenStack Shared File Systems service (manila), Ceph File System (CephFS), and Ceph through NFS.

The Shared File Systems service provides shares, where a share is an individual file system namespace and a unit of storage with a defined size. Shared file system storage inherently allows multiple clients to connect, read, and write data to any given share, but you must give each client access to the share through the Shared File Systems service access control APIs before they can connect.

With CephFS, a share is considered a directory with a defined quota and a layout that points to a particular storage pool or namespace. CephFS quotas limit the size of a directory to the size share that the Shared File Systems service creates. Access to CephFS through NFS shares is provided by specifying the IP address of the client.

With CephFS through NFS, file shares are provisioned and accessed through the NFS protocol. The NFS protocol also handles security.

2.1.3. Isolated network used by CephFS through NFS
Copia collegamento

CephFS through NFS deployments use an extra isolated network, StorageNFS. This network is deployed so users can mount shares over NFS on that network without accessing the Storage or Storage Management networks which are reserved for infrastructure traffic.

For more information about isolating networks, see Basic network isolation in the Director Installation and Usage guide.

To install CephFS through NFS, complete the following procedures:

  1. Install the ceph-ansible package. See Section 2.2.1, “Installing the ceph-ansible package”
  2. Generate the custom roles file, roles_data.yaml, and network_data.yaml file. See Section 2.2.1.1, “Generating the custom roles file”
  3. Deploy Ceph, the Shared File Systems service (manila), and CephFS by using the openstack overcloud deploy command with custom roles and environments. See Section 2.2.2, “Deploying the updated environment”
  4. Configure the isolated StorageNFS network and create the default share type. See Section 2.2.3, “Completing post-deployment configuration”

Examples use the standard stack user in the Red Hat Platform (RHOSP) environment.

Perform tasks as part of a RHOSP installation or environment update.

2.2.1. Installing the ceph-ansible package
Copia collegamento

Install the ceph-ansible package to be installed on an undercloud node to deploy containerized Ceph.

Procedure

  1. Log in to an undercloud node as the stack user.

  2. Install the ceph-ansible package: 

    [stack@undercloud-0 ~]$ sudo dnf install -y ceph-ansible
[stack@undercloud-0 ~]$ sudo dnf list ceph-ansible
...
Installed Packages
ceph-ansible.noarch    4.0.23-1.el8cp      @rhelosp-ceph-4-tools
    Copy to Clipboard Toggle word wrap

2.2.1.1. Generating the custom roles file
Copia collegamento

The ControllerStorageNFS custom role configures the isolated StorageNFS network. This role is similar to the default Controller.yaml role file with the addition of the StorageNFS network and the CephNfs service, indicated by the OS::TripleO::Services:CephNfs command. 

[stack@undercloud ~]$ cd /usr/share/openstack-tripleo-heat-templates/roles
[stack@undercloud roles]$ diff Controller.yaml ControllerStorageNfs.yaml
16a17
> 	- StorageNFS
50a45
> 	- OS::TripleO::Services::CephNfs
Copy to Clipboard Toggle word wrap

For more information about the openstack overcloud roles generate command, see Roles in the Advanced Overcloud Customization guide.

The openstack overcloud roles generate command creates a custom roles_data.yaml file including the services specified after -o. In the following example, the roles_data.yaml file created has the services for ControllerStorageNfs, Compute, and CephStorage.

Note

If you have an existing roles_data.yaml file, modify it to add ControllerStorageNfs, Compute, and CephStorage services to the configuration file. For more information, see Roles in the Advanced Overcloud Customization guide.

Procedure

  1. Log in to an undercloud node as the stack user,

  2. Use the openstack overcloud roles generate command to create the roles_data.yaml file: 

    [stack@undercloud ~]$ openstack overcloud roles generate --roles-path /usr/share/openstack-tripleo-heat-templates/roles -o /home/stack/roles_data.yaml ControllerStorageNfs Compute CephStorage
    Copy to Clipboard Toggle word wrap

2.2.2. Deploying the updated environment
Copia collegamento

When you are ready to deploy your environment, use the openstack overcloud deploy command with the custom environments and roles required to run CephFS with NFS-Ganesha.

The overcloud deploy command has the following options in addition to other required options.

Expand
ActionOptionAdditional information

Add the extra StorageNFS network with network_data_ganesha.yaml

-n /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml

Section 2.2.2.1, “StorageNFS and network_data_ganesha.yaml file”

Add the custom roles defined in roles_data.yaml file from the previous section

-r /home/stack/roles_data.yaml

Section 2.2.1.1, “Generating the custom roles file”

Deploy the Ceph daemons with ceph-ansible.yaml

-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml

Initiating Overcloud Deployment in the Deploying an Overcloud with Containerized Red Hat Ceph guide.

Deploy the Ceph metadata server with ceph-mds.yaml

-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-mds.yaml

Initiating Overcloud Deployment in the Deploying an Overcloud with Containerized Red Hat Ceph guide

Deploy the Shared File Systems (manila) service with the CephFS through NFS back end. Configure NFS-Ganesha with director.

-e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml

Section 2.2.2.2, “The CephFS back-end environment file”

The following example shows an openstack overcloud deploy command with options to deploy CephFS through NFS-Ganesha, Ceph cluster, Ceph MDS, and the isolated StorageNFS network: 

[stack@undercloud ~]$ openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates  \
-n /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml \
-r /home/stack/roles_data.yaml \
-e /home/stack/containers-default-parameters.yaml   \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml   \
-e /home/stack/network-environment.yaml  \
-e/usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml  \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-mds.yaml  \
-e /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml
Copy to Clipboard Toggle word wrap

For more information about the openstack overcloud deploy command, see Deployment command in the Director Installation and Usage guide.

2.2.2.1. StorageNFS and network_data_ganesha.yaml file
Copia collegamento

Use composable networks to define custom networks and assign them to any role. Instead of using the standard network_data.yaml file, you can configure the StorageNFS composable network with the network_data_ganesha.yaml file. Both of these roles are available in the /usr/share/openstack-tripleo-heat-templates directory.

The network_data_ganesha.yaml file contains an additional section that defines the isolated StorageNFS network. Although the default settings work for most installations, you must edit the YAML file to add your network settings, including the VLAN ID, subnet, and other settings. 

name: StorageNFS
enabled: true
vip: true
name_lower: storage_nfs
vlan: 70
ip_subnet: '172.17.0.0/20'
allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]
ipv6_subnet: 'fd00:fd00:fd00:7000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:7000::4', 'end': 'fd00:fd00:fd00:7000::fffe'}]
Copy to Clipboard Toggle word wrap

For more information about composable networks, see Using Composable Networks in the Advanced Overcloud Customization guide.

2.2.2.2. The CephFS back-end environment file
Copia collegamento

The integrated environment file for defining a CephFS back end, manila-cephfsganesha-config.yaml, is located in /usr/share/openstack-tripleo-heat-templates/environments/.

The manila-cephfsganesha-config.yaml environment file contains settings relevant to the deployment of the Shared File Systems service (manila). The back-end default settings work for most environments. The following example shows the default values that director uses during deployment of the Shared File Systems service: 

[stack@undercloud ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml
# A Heat environment file which can be used to enable a
# a Manila CephFS-NFS driver backend.
resource_registry:
  OS::TripleO::Services::ManilaApi: ../deployment/manila/manila-api-container-puppet.yaml
  OS::TripleO::Services::ManilaScheduler: ../deployment/manila/manila-scheduler-container-puppet.yaml
  # Only manila-share is pacemaker managed:
  OS::TripleO::Services::ManilaShare: ../deployment/manila/manila-share-pacemaker-puppet.yaml
  OS::TripleO::Services::ManilaBackendCephFs: ../deployment/manila/manila-backend-cephfs.yaml
  # ceph-nfs (ganesha) service is installed and configured by ceph-ansible
  # but it's still managed by pacemaker
  OS::TripleO::Services::CephNfs: ../deployment/ceph-ansible/ceph-nfs.yaml

parameter_defaults:
  ManilaCephFSBackendName: cephfs
1 

  ManilaCephFSDriverHandlesShareServers: false
2 

  ManilaCephFSCephFSAuthId: 'manila'
3 

  ManilaCephFSCephFSEnableSnapshots: false
4 

  # manila cephfs driver supports either native cephfs backend - 'CEPHFS'
  # (users mount shares directly from ceph cluster), or nfs-ganesha backend -
  # 'NFS' (users mount shares through nfs-ganesha server)
  ManilaCephFSCephFSProtocolHelperType: 'NFS'
Copy to Clipboard Toggle word wrap

The parameter_defaults header signifies the start of the configuration. In this section, you can edit settings to override default values set in resource_registry. This includes values set by OS::Tripleo::Services::ManilaBackendCephFs, which sets defaults for a CephFS back end.

1
ManilaCephFSBackendName sets the name of the manila configuration of your CephFS back end. In this case, the default back-end name is cephfs.
2
ManilaCephFSDriverHandlesShareServers controls the lifecycle of the share server. When set to false, the driver does not handle the lifecycle. This is the only supported option.
3
ManilaCephFSCephFSAuthId defines the Ceph auth ID that director creates for the manila service to access the Ceph cluster.
4
ManilaCephFSCephFSEnableSnapshots controls snapshot activation. The false value indicates that snapshots are not enabled. This feature is currently not supported.

For more information about environment files, see Environment Files in the Director Installation and Usage guide.

2.2.3. Completing post-deployment configuration
Copia collegamento

You must complete two post-deployment configuration tasks before you create NFS shares, grant user access, and mount NFS shares.

  • Map the Networking service (neutron) StorageNFS network to the isolated data center Storage NFS network.
  • Create the default share type.

After you complete these steps, the tenant compute instances can create, allow access to, and mount NFS shares.

2.2.3.1. Creating the storage provider network
Copia collegamento

You must map the new isolated StorageNFS network to a Networking (neutron) provider network. The Compute VMs attach to the network to access share export locations that are provided by the NFS-Ganesha gateway.

For information about network security with the Shared File Systems service, see Hardening the Shared File Systems Service in the Security and Hardening Guide.

Procedure

The openstack network create command defines the configuration for the StorageNFS neutron network.

  1. From an undercloud node, enter the following command: 

    [stack@undercloud ~]$ source ~/overcloudrc
    Copy to Clipboard Toggle word wrap

  2. On an undercloud node, create the StorageNFS network: 

    (overcloud) [stack@undercloud-0 ~]$ openstack network create StorageNFS --share  --provider-network-type vlan --provider-physical-network datacentre --provider-segment 70
    Copy to Clipboard Toggle word wrap

    You can enter this command with the following options:

    • For the --provider-physical-network option, use the default value datacentre, unless you set another tag for the br-isolated bridge through NeutronBridgeMappings in your tripleo-heat-templates.
    • For the --provider-segment option, use the VLAN value set for the StorageNFS isolated network in the heat template, /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml. This value is 70, unless the deployer modified the isolated network definitions.
    • For the --provider-network-type option, use the value vlan.

2.2.3.2. Configuring the shared provider StorageNFS network
Copia collegamento

Create a corresponding StorageNFSSubnet on the neutron-shared provider network. Ensure that the subnet is the same as the storage_nfs network definition in the network_data.yml file and ensure that the allocation range for the StorageNFS subnet and the corresponding undercloud subnet do not overlap. No gateway is required because the StorageNFS subnet is dedicated to serving NFS shares.

Prerequisites

  • The start and ending IP range for the allocation pool.
  • The subnet IP range.
2.2.3.2.1. Configuring the shared provider StorageNFS IPv4 network
Copia collegamento

Procedure

  1. Log in to an overcloud node.
  2. Source your overcloud credentials.

  3. Use the example command to provision the network and make the following updates:

    1. Replace the start=172.17.0.4,end=172.17.0.250 IP values with the IP values for your network.
    2. Replace the 172.17.0.0/20 subnet range with the subnet range for your network. 
[stack@undercloud-0 ~]$ openstack subnet create --allocation-pool start=172.17.0.4,end=172.17.0.250 --dhcp --network StorageNFS --subnet-range 172.17.0.0/20 --gateway none StorageNFSSubnet
Copy to Clipboard Toggle word wrap
2.2.3.2.2. Configuring the shared provider StorageNFS IPv6 network
Copia collegamento

Procedure

  1. Log in to an overcloud node.

  2. Use the sample command to provision the network, updating values as needed.

    • Replace the fd00:fd00:fd00:7000::/64 subnet range with the subnet range for your network. 
[stack@undercloud-0 ~]$ openstack subnet create --ip-version 6 --dhcp --network StorageNFS --subnet-range fd00:fd00:fd00:7000::/64 --gateway none --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful StorageNFSSubnet -f yaml
Copy to Clipboard Toggle word wrap

2.2.3.3. Configuring a default share type
Copia collegamento

You can use the Shared File Systems service to define share types that you can use to create shares with specific settings. Share types work like Block Storage volume types. Each type has associated settings, for example, extra specifications. When you invoke the type during share creation the settings apply to the shared file system.

With Red Hat OpenStack Platform (RHOSP) director, you must create a default share type before you open the cloud for users to access. For CephFS with NFS, use the manila type-create command: 

$ manila type-create default false
Copy to Clipboard Toggle word wrap

For more information about share types, see Creating a share type in the Storage Guide.

Torna in cima
Red Hat logoGithubredditYoutubeTwitter

Formazione

Prova, acquista e vendi

Community

Informazioni sulla documentazione di Red Hat

Aiutiamo gli utenti Red Hat a innovarsi e raggiungere i propri obiettivi con i nostri prodotti e servizi grazie a contenuti di cui possono fidarsi. Esplora i nostri ultimi aggiornamenti.

Rendiamo l’open source più inclusivo

Red Hat si impegna a sostituire il linguaggio problematico nel codice, nella documentazione e nelle proprietà web. Per maggiori dettagli, visita il Blog di Red Hat.

Informazioni su Red Hat

Forniamo soluzioni consolidate che rendono più semplice per le aziende lavorare su piattaforme e ambienti diversi, dal datacenter centrale all'edge della rete.

Theme

© 2025 Red Hat