Chapter 11. Management of NFS-Ganesha gateway using the Ceph Orchestrator (Limited Availability)

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the mgr/nfs module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable nfs

3. Create the cluster:

   Syntax

   ceph nfs cluster create CLUSTER_NAME ["HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"]

   The CLUSTER_NAME is an arbitrary string and HOST_NAME_1 is an optional string signifying the hosts to deploy NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph nfs cluster create nfsganesha "host01 host02"
   NFS Cluster Created Successful

   This creates an NFS_Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the RADOS pool namespace, and enable the application. For RBD pools, enable RBD.

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME freeform/rgw/rbd/cephfs/nfs
   rbd pool init -p POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha nfs
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

3. Deploy NFS-Ganesha gateway using placement specification in the command line interface:

   Syntax

   ceph orch apply nfs SERVICE_ID --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"

   Example

   [ceph: root@host01 /]# ceph orch apply nfs foo --placement="2 host01 host02"

   This deploys an NFS-Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

2. Edit the nfs.yaml specification file.

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2

   Example

   # cat nfs.yaml
   service_type: nfs
   service_id: foo
   placement:
     hosts:
       - host01
       - host02

   • Optional: Enable the NLM to support locking for NFS protocol v3 support by adding enable_nlm: true in the ganesha.yaml specification file.

     Syntax

     service_type: nfs
     service_id: SERVICE_ID
     placement:
       hosts:
         - HOST_NAME_1
         - HOST_NAME_2
     spec:
         enable_nlm: true

     Example

     # cat ganesha.yaml
     service_type: nfs
     service_id: foo
     placement:
       hosts:
         - host01
         - host02
     spec:
         enable_nlm: true

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

4. Create the RADOS pool, namespace, and enable RBD:

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME rbd
   rbd pool init -p POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha rbd
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

5. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

6. Deploy NFS-Ganesha gateway using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the NFS cluster with the --ingress flag:

   Syntax

   ceph nfs cluster create CLUSTER_ID [PLACEMENT] [--port PORT_NUMBER] [--ingress --virtual-ip IP_ADDRESS/CIDR_PREFIX]

   • Replace CLUSTER_ID with a unique string to name the NFS Ganesha cluster.
   • Replace PLACEMENT with the number of NFS servers to deploy and the host or hosts that you want to deploy the NFS Ganesha daemon containers on.

   • Use the --port PORT_NUMBER flag to deploy NFS on a port other than the default port of 12049.

     Note

     With ingress mode, the high-availability proxy takes port 2049 and NFS services are deployed on 12049.

   • The --ingress flag combined with the --virtual-ip flag, deploys NFS with a high-availability front-end (virtual IP and load balancer).

   • Replace --virtual-ip IP_ADDRESS with an IP address to provide a known, stable NFS endpoint that all clients can use to mount NFS exports. The --virtual-ip must include a CIDR prefix length. The virtual IP will normally be configured on the first identified network interface that has an existing IP in the same subnet.

     Note

     The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter. In the below example, one active NFS server is requested and two hosts are allocated.

     Example

     [ceph: root@host01 /]# ceph nfs cluster create mycephnfs "1 host02 host03" --ingress --virtual-ip 10.10.128.75/22

     Note

     Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

3. Check that the services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID
   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.mycephnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.mycephnfs  ?:12049      2/2  0s ago     20s  host02;host03
   
   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mycephnfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mycephnfs  10.10.128.75:2049,9049      4/4   46s ago    73s  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List existing NFS clusters:

   Example

   [ceph: root@host01 /]# ceph nfs cluster ls
   
   mynfs

   Note

   If a standalone NFS cluster is created on one node, you need to increase it to two or more nodes for HA. To increase the NFS service, edit the nfs.yaml file and increase the placements with the same port number.

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Example

   service_type: nfs
   service_id: mynfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   In this example, the existing NFS service is running on port 12345 and an additional node is added to the NFS cluster with the same port.

   Apply the nfs.yaml service specification changes to upgrade to a two node NFS service:

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

3. Edit the ingress.yaml specification file with the existing NFS cluster ID:

   Syntax

   service_type: SERVICE_TYPE
   service_id: SERVICE_ID
   placement:
     count: PLACEMENT
   spec:
     backend_service: SERVICE_ID_BACKEND 1
     frontend_port: FRONTEND_PORT
     monitor_port: MONITOR_PORT 2
     virtual_ip: VIRTUAL_IP_WITH_CIDR

   1

   SERVICE_ID_BACKEND should include a PORT property that is not 2049 to avoid conflicting with the ingress service, which could be placed on the same host.

   2

   MONITOR_PORT is used to access the haproxy load status page.

   Example

   service_type: ingress
   service_id: nfs.mynfs
   placement:
     count: 2
   spec:
     backend_service: nfs.mynfs
     frontend_port: 2049
     monitor_port: 9000
     virtual_ip: 10.10.128.75/22

4. Deploy the ingress service:

   Example

   [ceph: root@host01 /]# ceph orch apply -i ingress.yaml

   Note

   Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

5. Check that the ingress services have successfully started:

   Syntax

   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mynfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mynfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Ensure the NFS module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls | more

3. Exit out of the Cephadm shell and create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

4. Edit the nfs.yaml file to include the following details:

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Note

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Example

   service_type: nfs
   service_id: cephfsnfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   In this example, the server is run on the non-default port of 12345, instead of the default port of 2049, on host02 and host03.

5. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

6. Log into the Cephadm shell and navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

7. Deploy the NFS service using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

   Note

   Deployment of the NFS service is asynchronous and the command might return before the services have completely started.

8. Check that the NFS services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.cephfsnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.cephfsnfs  ?:12345      2/2  3m ago     13m  host02;host03

9. Exit out of the Cephadm shell and create the ingress.yaml file:

   Example

   [root@host01 ~]# touch ingress.yaml

10. Edit the ingress.yaml file to include the following details:

    Syntax

    service_type: SERVICE_TYPE
    service_id: SERVICE_ID
    placement:
      count: PLACEMENT
    spec:
      backend_service: SERVICE_ID_BACKEND
      frontend_port: FRONTEND_PORT
      monitor_port: MONITOR_PORT
      virtual_ip: VIRTUAL_IP_WITH_CIDR

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      count: 2
    spec:
      backend_service: nfs.cephfsnfs
      frontend_port: 2049
      monitor_port: 9000
      virtual_ip: 10.10.128.75/22

    Note

    In this example, placement: count: 2 deploys the keepalived and haproxy service on random nodes. To specify the nodes to deploy keepalived and haproxy on, use the placement: hosts option:

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      hosts:
        - host02
        - host03

11. Mount the YAML file under a directory in the container:

    Example

    [root@host01 ~]# cephadm shell --mount ingress.yaml:/var/lib/ceph/ingress.yaml

12. Log into the Cephadm shell and navigate to the directory:

    Example

    [ceph: root@host01 /]# cd /var/lib/ceph/

13. Deploy the ingress service using service specification:

    Syntax

    ceph orch apply -i FILE_NAME.yaml

    Example

    [ceph: root@host01 ceph]# ceph orch apply -i ingress.yaml

14. Check that the ingress services have successfully started:

    Syntax

    ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

    Example

    [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.cephfsnfs
    
    NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
    ingress.nfs.cephfsnfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Update the cluster:

   Syntax

   ceph orch apply nfs CLUSTER_NAME ["HOST_NAME_1,HOST_NAME_2,HOST_NAME_3"]

   The CLUSTER_NAME is an arbitrary string, HOST_NAME_1 is an optional string signifying the hosts to update the deployed NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph orch apply nfs nfsganesha "host02"

   This updates the nfsganesha cluster on host02.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. View the NFS-Ganesha cluster information:

   Syntax

   ceph nfs cluster info CLUSTER_NAME

   Example

   [ceph: root@host01 /]# ceph nfs cluster info nfsganesha
   
   {
       "nfsganesha": [
           {
               "hostname": "host02",
               "ip": [
                   "10.10.128.70"
               ],
               "port": 2049
           }
       ]
   }

Procedure

1. As a root user, fetch the FSID of the storage cluster:

   Example

   [root@host03 ~]# cephadm ls

   Copy the FSID and the name of the service.

2. Fetch the logs:

   Syntax

   cephadm logs --fsid FSID --name SERVICE_NAME

   Example

   [root@host03 ~]# cephadm logs --fsid 499829b4-832f-11eb-8d6d-001a4a000635 --name nfs.foo.host03

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. The following is an example of the default configuration of NFS-Ganesha cluster:

   Example

   # {{ cephadm_managed }}
   NFS_CORE_PARAM {
           Enable_NLM = false;
           Enable_RQUOTA = false;
           Protocols = 4;
   }
   
   MDCACHE {
           Dir_Chunk = 0;
   }
   
   EXPORT_DEFAULTS {
           Attr_Expiration_Time = 0;
   }
   
   NFSv4 {
           Delegations = false;
           RecoveryBackend = 'rados_cluster';
           Minor_Versions = 1, 2;
   }
   
   RADOS_KV {
           UserId = "{{ user }}";
           nodeid = "{{ nodeid}}";
           pool = "{{ pool }}";
           namespace = "{{ namespace }}";
   }
   
   RADOS_URLS {
           UserId = "{{ user }}";
           watch_url = "{{ url }}";
   }
   
   RGW {
           cluster = "ceph";
           name = "client.{{ rgw_user }}";
   }
   
   %url    {{ url }}

3. Customize the NFS-Ganesha cluster configuration. The following are two examples for customizing the configuration:

   • Change the log level:

     Example

     LOG {
      COMPONENTS {
          ALL = FULL_DEBUG;
      }
     }

   • Add custom export block:

     1. Create the user.

        Note

        User specified in FSAL blocks should have proper caps for NFS-Ganesha daemons to access the Ceph cluster.

        Syntax

        ceph auth get-or-create client.USER_ID mon 'allow r' osd 'allow rw pool=.nfs namespace=NFS_CLUSTER_NAME, allow rw tag cephfs data=FS_NAME' mds 'allow rw path=EXPORT_PATH'

        Example

        [ceph: root@host01 /]# ceph auth get-or-create client.f64f341c-655d-11eb-8778-fa163e914bcc mon 'allow r' osd 'allow rw pool=.nfs namespace=nfs_cluster_name, allow rw tag cephfs data=fs_name' mds 'allow rw path=export_path'

     2. Navigate to the following directory:

        Syntax

        cd /var/lib/ceph/DAEMON_PATH/

        Example

        [ceph: root@host01 /]# cd /var/lib/ceph/nfs/

        If the nfs directory does not exist, create a directory in the path.

     3. Create a new configuration file:

        Syntax

        touch PATH_TO_CONFIG_FILE

        Example

        [ceph: root@host01 nfs]#  touch nfs-ganesha.conf

     4. Edit the configuration file by adding the custom export block. It creates a single export and that is managed by the Ceph NFS export interface.

        Syntax

        EXPORT {
          Export_Id = NUMERICAL_ID;
          Transports = TCP;
          Path = PATH_WITHIN_CEPHFS;
          Pseudo = BINDING;
          Protocols = 4;
          Access_Type = PERMISSIONS;
          Attr_Expiration_Time = 0;
          Squash = None;
          FSAL {
            Name = CEPH;
            Filesystem = "FILE_SYSTEM_NAME";
            User_Id = "USER_NAME";
            Secret_Access_Key = "USER_SECRET_KEY";
          }
        }

        Example

        EXPORT {
          Export_Id = 100;
          Transports = TCP;
          Path = /;
          Pseudo = /ceph/;
          Protocols = 4;
          Access_Type = RW;
          Attr_Expiration_Time = 0;
          Squash = None;
          FSAL {
            Name = CEPH;
            Filesystem = "filesystem name";
            User_Id = "user id";
            Secret_Access_Key = "secret key";
          }
        }

4. Apply the new configuration the cluster:

   Syntax

   ceph nfs cluster config set CLUSTER_NAME -i PATH_TO_CONFIG_FILE

   Example

   [ceph: root@host01 nfs]# ceph nfs cluster config set nfs-ganesha -i /var/lib/ceph/nfs/nfs-ganesha.conf

   This also restarts the service for the custom configuration.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Reset the NFS-Ganesha configuration:

   Syntax

   ceph nfs cluster config reset CLUSTER_NAME

   Example

   [ceph: root@host01 /]# ceph nfs cluster config reset nfs-cephfs

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Delete the cluster:

   Syntax

   ceph nfs cluster rm CLUSTER_NAME

   The CLUSTER_NAME is an arbitrary string.

   Example

   [ceph: root@host01 /]# ceph nfs cluster rm nfsganesha
   NFS Cluster Deleted Successfully

   Note

   The delete option is deprecated and you need to use rm to delete an NFS cluster.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the service:

   Example

   [ceph: root@host01 /]# ceph orch ls

3. Remove the service:

   Syntax

   ceph orch rm SERVICE_NAME

   Example

   [ceph: root@host01 /]# ceph orch rm nfs.foo