CephFS via NFS Back End Guide for the Shared File System Service

In addition to the monitor, OSD, Rados Gateway (RGW), and manager services deployed when Ceph provides object and/or block storage, a Ceph metadata service (MDS) is required for CephFS and an NFS-Ganesha service is required as a gateway to native CephFS using the NFS protocol. (For user-facing object storage, an RGW service is also deployed). The gateway runs the CephFS client to access the Ceph public network and is under administrative rather than end-user control.

NFS-Ganesha runs in its own docker container that interfaces both to the Ceph public network and to a new isolated network, StorageNFS. OpenStack director’s composable network feature is used to deploy this network and connect it to the controller nodes. The cloud administrator then configures the network as a neutron provider network.

NFS-Ganesha accesses CephFS over the Ceph public network and binds its NFS service using an address on the StorageNFS network.

To access NFS shares, user VMs (nova instances) are provisioned with an additional NIC that connects to the Storage NFS network. Export locations for CephFS shares appear as standard NFS IP:<path> tuples using the NFS-Ganesha server’s VIP on the StorageNFS network. Access control for user VMs is done using the user VM’s IP on that network.

Neutron security groups prevent the user VM belonging to tenant 1 from accessing a user VM belonging to tenant 2 over the StorageNFS network. Tenants share the same CephFS filesystem but tenant data path separation is enforced because user VMs can only access files under export trees: /path/to/share1/…., /path/to/share2/….

When OpenStack director starts the Ceph service daemons, they manage their own high availability (HA) state and, in general, there are multiple instances of these daemons running. By contrast, in this release, only one instance of NFS-Ganesha can serve file shares at a time.

To avoid a single point of failure in the data path for CephFS via NFS shares, NFS-Ganesha runs on an OpenStack controller node in an active-passive configuration managed by a Pacemaker-Corosync cluster. NFS-Ganesha acts across the controller nodes as a virtual service with a virtual service IP address.

If a controller fails (or the service on a particular controller node fails and cannot be recovered on that node) Pacemaker-Corosync starts a new NFS-Ganesha instance on a different controller using the same virtual IP. Existing client mounts are preserved because they use the virtual IP for the export location of shares.

Using default NFS mount-option settings and NFS 4.1 or greater, after a failure, TCP connections are reset and clients reconnect. I/O operations temporarily stop responding during failover, but they will not fail. Application I/O also stops responding, but resumes after failover completes.

New connections, new lock-state, and so forth are refused until after a grace period of up to 90 seconds during which the server waits for clients to reclaim their locks. NFS-Ganesha keeps a list of the clients and will exit the grace period earlier, if it sees that all clients reclaimed their locks.

The ControllerStorageNFS custom role is used to set up the isolated StorageNFS network. This role is similar to the default Controller.yaml role file with the addition of the StorageNFS network and the CephNfs service (indicated by OS::TripleO::Services:CephNfs).

[stack@undercloud ~]$ cd /usr/share/openstack-tripleo-heat-templates/roles
[stack@undercloud roles]$ diff Controller.yaml ControllerStorageNfs.yaml
16a17
> 	- StorageNFS
50a45
> 	- OS::TripleO::Services::CephNfs

For information about the openstack overcloud roles generate command, refer to the Roles section of the Advanced Overcloud Customization Guide.

Composable networks let you define custom networks and assign them to any role. Instead of using the standard network_data.yaml file, the StorageNFS composable network is configured using the network_data_ganesha.yaml file. Both of these roles are available in the /usr/share/openstack-tripleo-heat-templates directory.

The network_data_ganesha.yaml file contains an additional section that defines the isolated StorageNFS network. While the default settings will work for most installations, you will still need to edit the YAML file to add your network settings, including the VLAN ID, subnet, etc.

name: StorageNFS
enabled: true
vip: true
ame_lower: storage_nfs
vlan: 70
ip_subnet: '172.16.4.0/24'
allocation_pools: [{'start': '172.16.4.4', 'end': '172.16.4.250'}]
ipv6_subnet: 'fd00:fd00:fd00:7000::/64'
ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:7000::10', 'end': 'fd00:fd00:fd00:7000:ffff:ffff:ffff:fffe'}]

For more information on composable networks, refer to the Using Composable Networks section in the Advanced Overcloud Customization Guide.

The integrated environment file for defining a CephFS back end is located in the following path of an undercloud node:

/usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml

The manila-cephfsganesha-config.yaml environment file contains settings relevant to the deployment of the Shared File System service. The back end default settings should work for most environments. The example shows the default values used by the director when deploying the Shared File System service:

[stack@undercloud ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/manila-cephfsganesha-config.yaml
# A Heat environment file which can be used to enable a
# a Manila CephFS-NFS driver backend.
resource_registry:
  OS::TripleO::Services::ManilaApi: ../docker/services/manila-api.yaml
  OS::TripleO::Services::ManilaScheduler: ../docker/services/manila-scheduler.yaml
  # Only manila-share is pacemaker managed:
  OS::TripleO::Services::ManilaShare: ../docker/services/pacemaker/manila-share.yaml
  OS::TripleO::Services::ManilaBackendCephFs: ../puppet/services/manila-backend-cephfs.yaml
  # ceph-nfs (ganesha) service is installed and configured by ceph-ansible
  # but it's still managed by pacemaker
  OS::TripleO::Services::CephNfs: ../docker/services/ceph-ansible/ceph-nfs.yaml

parameter_defaults:
  ManilaCephFSBackendName: cephfs 1
  ManilaCephFSDriverHandlesShareServers: false 2
  ManilaCephFSCephFSAuthId: 'manila' 3
  ManilaCephFSCephFSEnableSnapshots: false 4
  # manila cephfs driver supports either native cephfs backend - 'CEPHFS'
  # (users mount shares directly from ceph cluster), or nfs-ganesha backend -
  # 'NFS' (users mount shares through nfs-ganesha server)
  ManilaCephFSCephFSProtocolHelperType: 'NFS'

The parameter_defaults header signifies the start of the configuration. Specifically, settings under this header let you override default values set in resource_registry. This includes values set by OS::Tripleo::Services::ManilaBackendCephFs, which sets defaults for a CephFS back end.

For more information about environment files, see Environment Files in the Advanced Overcloud Customization guide.

The new isolated StorageNFS network must be mapped to a neutron-shared provider network. The Compute VMs will attach to this neutron network to access share export locations provided by the NFS-Ganesha gateway.

For general information about network security with the Shared File System service, refer to the section Hardening the Shared File System Service in the Security and Hardening Guide.

To use this command:

Create a corresponding StorageNFSSubnet on the neutron shared provider network. Ensure that the subnet is the same as for the storage_nfs_subnet in the undercloud but make sure that the allocation range for this subnet and that of the corresponding undercloud subnet do not overlap. No gateway is required since this subnet is dedicated to serving NFS shares.

[stack@undercloud-0 ~]$ openstack subnet create --allocation-pool start=172.16.4.150,end=172.16.4.250 --dhcp --network StorageNFS --subnet-range 172.16.4.0/24 --gateway none StorageNFSSubnet
+-------------------+--------------------------------------+
| Field         	  | Value
+-------------------+--------------------------------------+
| allocation_pools  | 172.16.4.150-172.16.4.250
| cidr            	| 172.16.4.0/24
| created_at    	  | 2018-09-17T21:22:14Z
| description   	  |
| dns_nameservers   |
| enable_dhcp   	  | True
| gateway_ip    	  | None
| host_routes   	  |
| id            	  | 8c696d06-76b7-4d77-a375-fd2e71e3e480
| ip_version    	  | 4
| ipv6_address_mode | None
| ipv6_ra_mode  	  | None
| name          	  | StorageNFSSubnet
| network_id      	| cd272981-0a5e-4f3d-83eb-d25473f5176e
| project_id      	| 3ca3408d545143629cd0ec35d34aea9c
| revision_number   | 0
| segment_id    	  | None
| service_types 	  |
| subnetpool_id 	  | None
| tags          	  |
| updated_at    	  | 2018-09-17T21:22:14Z
+-------------------+--------------------------------------+

The Shared File System service allows you to define share types that you can use to create shares with specific settings. Share types work like Block Storage volume types: each type has associated settings (namely, extra specifications), and invoking the type during share creation applies those settings.

The OpenStack director expects a default share type. This default share type has to be created prior to opening the cloud for users to access. For CephFS with NFS, use the manila type-create command:

manila type-create default false

For information about share types, refer to the section Create and Manage Share Types of the Storage Guide.