Chapter 4. Configuring the core RDMA subsystem

Procedure

1. Display the hardware address of the device ib0:

   # ip link show ib0
   8: ib0: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 65520 qdisc pfifo_fast state UP mode DEFAULT qlen 256
       link/infiniband 80:00:02:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:31:78:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

   The last eight bytes of the address are required to create a udev rule in the next step.

2. To configure a rule that renames the device with the 00:02:c9:03:00:31:78:f2 hardware address to mlx4_ib0, edit the /etc/udev/rules.d/70-persistent-ipoib.rules file and add an ACTION rule:

   ACTION=="add", SUBSYSTEM=="net", DRIVERS=="?*", ATTR{type}=="32", ATTR{address}=="?*00:02:c9:03:00:31:78:f2", NAME="mlx4_ib0"

3. Reboot the host:

   # reboot

Verification

1. Log in as a member of the rdma group after editing the /etc/security/limits.conf file.

   Note that Red Hat Enterprise Linux applies updated ulimit settings when the user logs in.

2. Use the ulimit -l command to display the limit:

   $ ulimit -l
   unlimited

   If the command returns unlimited, the user can pin an unlimited amount of memory.

Procedure

1. Edit the /etc/rdma/modules/rdma.conf file and uncomment the modules that you want to enable:

   # These modules are loaded by the system if any RDMA devices is installed
   
   # iSCSI over RDMA client support
   ib_iser
   
   # iSCSI over RDMA target support
   ib_isert
   
   # SCSI RDMA Protocol target driver
   ib_srpt
   
   # User access to RDMA verbs (supports libibverbs)
   ib_uverbs
   
   # User access to RDMA connection management (supports librdmacm)
   rdma_ucm
   
   # RDS over RDMA support
   # rds_rdma
   
   # NFS over RDMA client support
   xprtrdma
   
   # NFS over RDMA server support
   svcrdma

2. Restart the service to make the changes effective:

   # systemctl restart <rdma-load-modules@rdma.service>

Procedure

1. Install the rdma-core package:

   # dnf install rdma-core

2. If the package was already installed, verify that the xprtrdma and svcrdma modules in the /etc/rdma/modules/rdma.conf file are uncommented:

   # NFS over RDMA client support
   xprtrdma
   # NFS over RDMA server support
   svcrdma

3. Optional. By default, NFS over RDMA uses port 20049. If you want to use a different port, set the rdma-port setting in the [nfsd] section of the /etc/nfs.conf file:

   rdma-port=<port>

4. Open the NFSoRDMA port in firewalld:

   # firewall-cmd --permanent --add-port={20049/tcp,20049/udp}
   # firewall-cmd --reload

   Adjust the port numbers if you set a different port than 20049.

5. Restart the nfs-server service:

   # systemctl restart nfs-server

Verification

1. On a client with InfiniBand hardware, perform the following steps:

   1. Install the following packages:

      # dnf install nfs-utils rdma-core

   2. Mount an exported NFS share over RDMA:

      # mount -o rdma server.example.com:/nfs/projects/ /mnt/

      If you set a port number other than the default (20049), pass port=<port_number> to the command:

      # mount -o rdma,port=<port_number> server.example.com:/nfs/projects/ /mnt/

   3. Verify that the share was mounted with the rdma option:

      # mount | grep "/mnt"
      server.example.com:/nfs/projects/ on /mnt type nfs (...,proto=rdma,...)