Chapter 10. Configuring NVMe over fabrics using NVMe/RDMA

Procedure

1. Create the nvmet-rdma subsystem:

   # modprobe nvmet-rdma
   
   # mkdir /sys/kernel/config/nvmet/subsystems/testnqn
   
   # cd /sys/kernel/config/nvmet/subsystems/testnqn

   Copy to Clipboard Toggle word wrap

   Replace testnqn with the subsystem name.

2. Allow any host to connect to this controller:

   # echo 1 > attr_allow_any_host

   Copy to Clipboard Toggle word wrap

3. Configure a namespace:

   # mkdir namespaces/10
   
   # cd namespaces/10

   Copy to Clipboard Toggle word wrap

   Replace 10 with the namespace number

4. Set a path to the NVMe device:

   # echo -n /dev/nvme0n1 > device_path

   Copy to Clipboard Toggle word wrap

5. Enable the namespace:

   # echo 1 > enable

   Copy to Clipboard Toggle word wrap

6. Create a directory with an NVMe port:

   # mkdir /sys/kernel/config/nvmet/ports/1
   
   # cd /sys/kernel/config/nvmet/ports/1

   Copy to Clipboard Toggle word wrap

7. Display the IP address of mlx5_ib0:

   # ip addr show mlx5_ib0
   
   8: mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP group default qlen 256
       link/infiniband 00:00:06:2f:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:e7:0f:f6 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
       inet 172.31.0.202/24 brd 172.31.0.255 scope global noprefixroute mlx5_ib0
          valid_lft forever preferred_lft forever
       inet6 fe80::e61d:2d03:e7:ff6/64 scope link noprefixroute
          valid_lft forever preferred_lft forever

   Copy to Clipboard Toggle word wrap

8. Set the transport address for the controller:

   # echo -n 172.31.0.202 > addr_traddr

   Copy to Clipboard Toggle word wrap

9. Set RDMA as the transport type:

   # echo rdma > addr_trtype
   
   # echo 4420 > addr_trsvcid

   Copy to Clipboard Toggle word wrap

10. Set the address family for the port:

    # echo ipv4 > addr_adrfam

    Copy to Clipboard Toggle word wrap

11. Create a soft link:

    # ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/testnqn

    Copy to Clipboard Toggle word wrap

Procedure

1. Install the nvmetcli package:

   # dnf install nvmetcli

   Copy to Clipboard Toggle word wrap

2. Download the rdma.json file:

   # wget http://git.infradead.org/users/hch/nvmetcli.git/blob_plain/0a6b088db2dc2e5de11e6f23f1e890e4b54fee64:/rdma.json

   Copy to Clipboard Toggle word wrap
3. Edit the rdma.json file and change the traddr value to 172.31.0.202.

4. Setup the controller by loading the NVMe controller configuration file:

   # nvmetcli restore rdma.json

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the nvme-cli tool:

   # dnf install nvme-cli

   Copy to Clipboard Toggle word wrap

2. Load the nvme-rdma module if it is not loaded:

   # modprobe nvme-rdma

   Copy to Clipboard Toggle word wrap

3. Discover available subsystems on the NVMe controller:

   # nvme discover -t rdma -a 172.31.0.202 -s 4420
   
   Discovery Log Number of Records 1, Generation counter 2
   =====Discovery Log Entry 0======
   trtype:  rdma
   adrfam:  ipv4
   subtype: nvme subsystem
   treq:    not specified, sq flow control disable supported
   portid:  1
   trsvcid: 4420
   subnqn:  testnqn
   traddr:  172.31.0.202
   rdma_prtype: not specified
   rdma_qptype: connected
   rdma_cms:    rdma-cm
   rdma_pkey: 0x0000

   Copy to Clipboard Toggle word wrap

4. Connect to the discovered subsystems:

   # nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420
   
   # lsblk
   NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   sda                            8:0    0 465.8G  0 disk
   ├─sda1                         8:1    0     1G  0 part /boot
   └─sda2                         8:2    0 464.8G  0 part
     ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
     ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
     └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
   nvme0n1
   
   # cat /sys/class/nvme/nvme0/transport
   rdma

   Copy to Clipboard Toggle word wrap

   Replace testnqn with the NVMe subsystem name.

   Replace 172.31.0.202 with the controller IP address.

   Replace 4420 with the port number.