Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 29. Overview of NVMe over fabric devices

Non-volatile Memory Express (NVMe) is an interface that allows host software utility to communicate with solid state drives. Use the following types of fabric transport to configure NVMe over fabric devices:
When using FC and RDMA, the solid-state drive does not have to be local to your system; it can be configured remotely through a FC or RDMA controller.

29.1. NVMe over fabrics using RDMA
Copy link

The following sections describe how to deploy an NVMe over RDMA (NVMe/RDMA) initiator configuration.

29.1.1. Configuring an NVMe over RDMA client
Copy link

Use this procedure to configure an NVMe/RDMA client using the NVMe management command line interface (nvme-cli).
  1. Install the nvme-cli package: 
    # yum install nvme-cli
    Copy to Clipboard Toggle word wrap
  2. Load the nvme-rdma module if it is not loaded: 
    # modprobe nvme-rdma
    Copy to Clipboard Toggle word wrap
  3. Discover available subsystems on the NVMe target: 
    # nvme discover -t rdma -a 172.31.0.202 -s 4420

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  testnqn
traddr:  172.31.0.202
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
    Copy to Clipboard Toggle word wrap
  4. Connect to the discovered subsystems: 
    # nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420

# lsblk

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 465.8G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0 464.8G  0 part
  ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
nvme0n1

# cat /sys/class/nvme/nvme0/transport

rdma
    Copy to Clipboard Toggle word wrap
    Replace testnqn with the NVMe subsystem name.
    Replace 172.31.0.202 with the target IP address.
    Replace 4420 with the port number.
  5. List the NVMe devices that are currently connected: 
    # nvme list
    Copy to Clipboard Toggle word wrap
  6. Optional: Disconnect from the target: 
    # nvme disconnect -n testnqn

NQN:testnqn disconnected 1 controller(s)

# lsblk

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 465.8G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0 464.8G  0 part
├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
└─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
    Copy to Clipboard Toggle word wrap

Additional resources

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat