Chapter 11. Management of NFS-Ganesha gateway using the Ceph Orchestrator (Limited Availability)

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Enable the mgr/nfs module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable nfs

   Copy to Clipboard Toggle word wrap

3. Create the cluster:

   Syntax

   ceph nfs cluster create CLUSTER_NAME ["HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"]

   Copy to Clipboard Toggle word wrap

   The CLUSTER_NAME is an arbitrary string and HOST_NAME_1 is an optional string signifying the hosts to deploy NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph nfs cluster create nfsganesha "host01 host02"
   NFS Cluster Created Successful

   Copy to Clipboard Toggle word wrap

   This creates an NFS_Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Create the RADOS pool namespace, and enable the application. For RBD pools, enable RBD.

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME freeform/rgw/rbd/cephfs/nfs
   rbd pool init -p POOL_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha nfs
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

   Copy to Clipboard Toggle word wrap

3. Deploy NFS-Ganesha gateway using placement specification in the command line interface:

   Syntax

   ceph orch apply nfs SERVICE_ID --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph orch apply nfs foo --placement="2 host01 host02"

   Copy to Clipboard Toggle word wrap

   This deploys an NFS-Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

   Copy to Clipboard Toggle word wrap

2. Edit the nfs.yaml file to include the following details:

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2

   Copy to Clipboard Toggle word wrap

   Example

   service_type: nfs
   service_id: foo
   placement:
     hosts:
       - host01
       - host02

   Copy to Clipboard Toggle word wrap

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

   Copy to Clipboard Toggle word wrap

4. Create the RADOS pool, namespace, and enable RBD:

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME rbd
   rbd pool init -p POOL_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha rbd
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

   Copy to Clipboard Toggle word wrap

5. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

   Copy to Clipboard Toggle word wrap

6. Deploy NFS-Ganesha gateway using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Create the NFS cluster with the --ingress flag:

   Syntax

   ceph nfs cluster create CLUSTER_ID [PLACEMENT] [--port PORT_NUMBER] [--ingress --virtual-ip IP_ADDRESS/CIDR_PREFIX]

   Copy to Clipboard Toggle word wrap

   • Replace CLUSTER_ID with a unique string to name the NFS Ganesha cluster.
   • Replace PLACEMENT with the number of NFS servers to deploy and the host or hosts that you want to deploy the NFS Ganesha daemon containers on.
   • Use the --port PORT_NUMBER flag to deploy NFS on a port other than the default port of 2049.
   • The --ingress flag combined with the --virtual-ip flag, deploys NFS with a high-availability front-end (virtual IP and load balancer).

   • Replace --virtual-ip IP_ADDRESS with an IP address to provide a known, stable NFS endpoint that all clients can use to mount NFS exports. The --virtual-ip must include a CIDR prefix length. The virtual IP will normally be configured on the first identified network interface that has an existing IP in the same subnet.

     Note

     The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter. In the below example, one active NFS server is requested and two hosts are allocated.

     Example

     [ceph: root@host01 /]# ceph nfs cluster create mycephnfs "1 host02 host03" --ingress --virtual-ip 10.10.128.75/22

     Copy to Clipboard Toggle word wrap

     Note

     Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

3. Check that the services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID
   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.mycephnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.mycephnfs  ?:12049      1/2  0s ago     20s  host02;host03
   
   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mycephnfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mycephnfs  10.10.128.75:2049,9049      4/4   46s ago    73s  count:2

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. List existing NFS clusters:

   Example

   [ceph: root@host01 /]# ceph nfs cluster ls
   
   mynfs

   Copy to Clipboard Toggle word wrap

   Note

   If a standalone NFS cluster is created on one node, you need to increase it to two or more nodes for HA. To increase the NFS service, edit the nfs.yaml file and increase the placements with the same port number.

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   service_type: nfs
   service_id: mynfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   Copy to Clipboard Toggle word wrap

   In this example, the existing NFS service is running on port 12345 and an additional node is added to the NFS cluster with the same port.

   Apply the nfs.yaml service specification changes to upgrade to a two node NFS service:

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

   Copy to Clipboard Toggle word wrap

3. Edit the ingress.yaml specification file with the existing NFS cluster ID:

   Syntax

   service_type: SERVICE_TYPE
   service_id: SERVICE_ID
   placement:
     count: PLACEMENT
   spec:
     backend_service: SERVICE_ID_BACKEND 

   1

   
     frontend_port: FRONTEND_PORT
     monitor_port: MONITOR_PORT 

   2

   
     virtual_ip: VIRTUAL_IP_WITH_CIDR

   Copy to Clipboard Toggle word wrap

   1

   SERVICE_ID_BACKEND should include a PORT property that is not 2049 to avoid conflicting with the ingress service, which could be placed on the same host.

   2

   MONITOR_PORT is used to access the haproxy load status page.

   Example

   service_type: ingress
   service_id: nfs.mynfs
   placement:
     count: 2
   spec:
     backend_service: nfs.mynfs
     frontend_port: 2049
     monitor_port: 9000
     virtual_ip: 10.10.128.75/22

   Copy to Clipboard Toggle word wrap

4. Deploy the ingress service:

   Example

   [ceph: root@host01 /]# ceph orch apply -i ingress.yaml

   Copy to Clipboard Toggle word wrap

   Note

   Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

5. Check that the ingress services have successfully started:

   Syntax

   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mynfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mynfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Ensure the NFS module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls | more

   Copy to Clipboard Toggle word wrap

3. Exit out of the Cephadm shell and create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

   Copy to Clipboard Toggle word wrap

4. Edit the nfs.yaml file to include the following details:

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Copy to Clipboard Toggle word wrap

   Note

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Example

   service_type: nfs
   service_id: cephfsnfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   Copy to Clipboard Toggle word wrap

   In this example, the server is run on the non-default port of 12345, instead of the default port of 2049, on host02 and host03.

5. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

   Copy to Clipboard Toggle word wrap

6. Log into the Cephadm shell and navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

   Copy to Clipboard Toggle word wrap

7. Deploy the NFS service using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

   Copy to Clipboard Toggle word wrap

   Note

   Deployment of the NFS service is asynchronous and the command might return before the services have completely started.

8. Check that the NFS services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.cephfsnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.cephfsnfs  ?:12345      2/2  3m ago     13m  host02;host03

   Copy to Clipboard Toggle word wrap

9. Exit out of the Cephadm shell and create the ingress.yaml file:

   Example

   [root@host01 ~]# touch ingress.yaml

   Copy to Clipboard Toggle word wrap

10. Edit the ingress.yaml file to include the following details:

    Syntax

    service_type: SERVICE_TYPE
    service_id: SERVICE_ID
    placement:
      count: PLACEMENT
    spec:
      backend_service: SERVICE_ID_BACKEND
      frontend_port: FRONTEND_PORT
      monitor_port: MONITOR_PORT
      virtual_ip: VIRTUAL_IP_WITH_CIDR

    Copy to Clipboard Toggle word wrap

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      count: 2
    spec:
      backend_service: nfs.cephfsnfs
      frontend_port: 2049
      monitor_port: 9000
      virtual_ip: 10.10.128.75/22

    Copy to Clipboard Toggle word wrap

    Note

    In this example, placement: count: 2 deploys the keepalived and haproxy service on random nodes. To specify the nodes to deploy keepalived and haproxy on, use the placement: hosts option:

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      hosts:
        - host02
        - host03

    Copy to Clipboard Toggle word wrap

11. Mount the YAML file under a directory in the container:

    Example

    [root@host01 ~]# cephadm shell --mount ingress.yaml:/var/lib/ceph/ingress.yaml

    Copy to Clipboard Toggle word wrap

12. Log into the Cephadm shell and navigate to the directory:

    Example

    [ceph: root@host01 /]# cd /var/lib/ceph/

    Copy to Clipboard Toggle word wrap

13. Deploy the ingress service using service specification:

    Syntax

    ceph orch apply -i FILE_NAME.yaml

    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 ceph]# ceph orch apply -i ingress.yaml

    Copy to Clipboard Toggle word wrap

14. Check that the ingress services have successfully started:

    Syntax

    ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.cephfsnfs
    
    NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
    ingress.nfs.cephfsnfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

    Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Update the cluster:

   Syntax

   ceph orch apply nfs CLUSTER_NAME ["HOST_NAME_1,HOST_NAME_2,HOST_NAME_3"]

   Copy to Clipboard Toggle word wrap

   The CLUSTER_NAME is an arbitrary string, HOST_NAME_1 is an optional string signifying the hosts to update the deployed NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph orch apply nfs nfsganesha "host02"

   Copy to Clipboard Toggle word wrap

   This updates the nfsganesha cluster on host02.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. View the NFS-Ganesha cluster information:

   Syntax

   ceph nfs cluster info CLUSTER_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph nfs cluster info nfsganesha
   
   {
       "nfsganesha": [
           {
               "hostname": "host02",
               "ip": [
                   "10.10.128.70"
               ],
               "port": 2049
           }
       ]
   }

   Copy to Clipboard Toggle word wrap

Procedure

1. As a root user, fetch the FSID of the storage cluster:

   Example

   [root@host03 ~]# cephadm ls

   Copy to Clipboard Toggle word wrap

   Copy the FSID and the name of the service.

2. Fetch the logs:

   Syntax

   cephadm logs --fsid FSID --name SERVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@host03 ~]# cephadm logs --fsid 499829b4-832f-11eb-8d6d-001a4a000635 --name nfs.foo.host03

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. The following is an example of the default configuration of NFS-Ganesha cluster:

   Example

   # {{ cephadm_managed }}
   NFS_CORE_PARAM {
           Enable_NLM = false;
           Enable_RQUOTA = false;
           Protocols = 4;
   }
   
   MDCACHE {
           Dir_Chunk = 0;
   }
   
   EXPORT_DEFAULTS {
           Attr_Expiration_Time = 0;
   }
   
   NFSv4 {
           Delegations = false;
           RecoveryBackend = 'rados_cluster';
           Minor_Versions = 1, 2;
   }
   
   RADOS_KV {
           UserId = "{{ user }}";
           nodeid = "{{ nodeid}}";
           pool = "{{ pool }}";
           namespace = "{{ namespace }}";
   }
   
   RADOS_URLS {
           UserId = "{{ user }}";
           watch_url = "{{ url }}";
   }
   
   RGW {
           cluster = "ceph";
           name = "client.{{ rgw_user }}";
   }
   
   %url    {{ url }}

   Copy to Clipboard Toggle word wrap

3. Customize the NFS-Ganesha cluster configuration. The following are two examples for customizing the configuration:

   • Change the log level:

     Example

     LOG {
      COMPONENTS {
          ALL = FULL_DEBUG;
      }
     }

     Copy to Clipboard Toggle word wrap

   • Add custom export block:

     1. Create the user.

        Note

        User specified in FSAL blocks should have proper caps for NFS-Ganesha daemons to access the Ceph cluster.

        Syntax

        ceph auth get-or-create client.USER_ID mon 'allow r' osd 'allow rw pool=.nfs namespace=NFS_CLUSTER_NAME, allow rw tag cephfs data=FS_NAME' mds 'allow rw path=EXPORT_PATH'

        Copy to Clipboard Toggle word wrap

        Example

        [ceph: root@host01 /]# ceph auth get-or-create client.f64f341c-655d-11eb-8778-fa163e914bcc mon 'allow r' osd 'allow rw pool=.nfs namespace=nfs_cluster_name, allow rw tag cephfs data=fs_name' mds 'allow rw path=export_path'

        Copy to Clipboard Toggle word wrap

     2. Navigate to the following directory:

        Syntax

        cd /var/lib/ceph/DAEMON_PATH/

        Copy to Clipboard Toggle word wrap

        Example

        [ceph: root@host01 /]# cd /var/lib/ceph/nfs/

        Copy to Clipboard Toggle word wrap

        If the nfs directory does not exist, create a directory in the path.

     3. Create a new configuration file:

        Syntax

        touch PATH_TO_CONFIG_FILE

        Copy to Clipboard Toggle word wrap

        Example

        [ceph: root@host01 nfs]#  touch nfs-ganesha.conf

        Copy to Clipboard Toggle word wrap

     4. Edit the configuration file by adding the custom export block. It creates a single export and that is managed by the Ceph NFS export interface.

        Syntax

        EXPORT {
          Export_Id = NUMERICAL_ID;
          Transports = TCP;
          Path = PATH_WITHIN_CEPHFS;
          Pseudo = BINDING;
          Protocols = 4;
          Access_Type = PERMISSIONS;
          Attr_Expiration_Time = 0;
          Squash = None;
          FSAL {
            Name = CEPH;
            Filesystem = "FILE_SYSTEM_NAME";
            User_Id = "USER_NAME";
            Secret_Access_Key = "USER_SECRET_KEY";
          }
        }

        Copy to Clipboard Toggle word wrap

        Example

        EXPORT {
          Export_Id = 100;
          Transports = TCP;
          Path = /;
          Pseudo = /ceph/;
          Protocols = 4;
          Access_Type = RW;
          Attr_Expiration_Time = 0;
          Squash = None;
          FSAL {
            Name = CEPH;
            Filesystem = "filesystem name";
            User_Id = "user id";
            Secret_Access_Key = "secret key";
          }
        }

        Copy to Clipboard Toggle word wrap

4. Apply the new configuration the cluster:

   Syntax

   ceph nfs cluster config set CLUSTER_NAME -i PATH_TO_CONFIG_FILE

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 nfs]# ceph nfs cluster config set nfs-ganesha -i /var/lib/ceph/nfs/nfs-ganesha.conf

   Copy to Clipboard Toggle word wrap

   This also restarts the service for the custom configuration.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Reset the NFS-Ganesha configuration:

   Syntax

   ceph nfs cluster config reset CLUSTER_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph nfs cluster config reset nfs-cephfs

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Delete the cluster:

   Syntax

   ceph nfs cluster rm CLUSTER_NAME

   Copy to Clipboard Toggle word wrap

   The CLUSTER_NAME is an arbitrary string.

   Example

   [ceph: root@host01 /]# ceph nfs cluster rm nfsganesha
   NFS Cluster Deleted Successfully

   Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. List the service:

   Example

   [ceph: root@host01 /]# ceph orch ls

   Copy to Clipboard Toggle word wrap

3. Remove the service:

   Syntax

   ceph orch rm SERVICE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph orch rm nfs.foo

   Copy to Clipboard Toggle word wrap