Chapter 29. Overview of NVMe over fabric devices
Non-volatile Memory Express (NVMe) is an interface that allows host software utility to communicate with solid state drives. Use the following types of fabric transport to configure
NVMe over fabric devices:
- NVMe over fabrics using Remote Direct Memory Access (RDMA). For information on how to configure NVMe/RDMA, see Section 29.1, “NVMe over fabrics using RDMA”.
- NVMe over fabrics using Fibre Channel (FC). For information on how to configure FC-NVMe, see Section 29.2, “NVMe over fabrics using FC”.
When using FC and RDMA, the solid-state drive does not have to be local to your system; it can be configured remotely through a FC or RDMA controller.
29.1. NVMe over fabrics using RDMA
The following sections describe how to deploy an NVMe over RDMA (NVMe/RDMA) initiator configuration.
29.1.1. Configuring an NVMe over RDMA client
Use this procedure to configure an NVMe/RDMA client using the NVMe management command line interface (
nvme-cli).
- Install the
nvme-clipackage:#yum install nvme-cli - Load the
nvme-rdmamodule if it is not loaded:#modprobe nvme-rdma - Discover available subsystems on the NVMe target:
#nvme discover -t rdma -a 172.31.0.202 -s 4420Discovery Log Number of Records 1, Generation counter 2 =====Discovery Log Entry 0====== trtype: rdma adrfam: ipv4 subtype: nvme subsystem treq: not specified, sq flow control disable supported portid: 1 trsvcid: 4420 subnqn: testnqn traddr: 172.31.0.202 rdma_prtype: not specified rdma_qptype: connected rdma_cms: rdma-cm rdma_pkey: 0x0000 - Connect to the discovered subsystems:
#nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420#lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home nvme0n1#cat /sys/class/nvme/nvme0/transportrdmaReplace testnqn with the NVMe subsystem name.Replace 172.31.0.202 with the target IP address.Replace 4420 with the port number. - List the NVMe devices that are currently connected:
#nvme list - Optional: Disconnect from the target:
#nvme disconnect -n testnqnNQN:testnqn disconnected 1 controller(s)#lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home
Additional resources
- For more information, see the
nvmeman page and the NVMe-cli Github repository.
