Search

Chapter 13. Configuring NVMe over fabrics using NVMe/TCP

download PDF

In an Non-volatile Memory Express™ (NVMe™) over TCP (NVMe/TCP) setup, the host mode is fully supported and the controller setup is not supported.

As a system administrator, complete the tasks in the following sections to deploy the NVMe/TCP setup:

Note

In Red Hat Enterprise Linux 9, the native NVMe multipathing is enabled by default. Enabling DM multipathing is not supported with NVMe/TCP.

13.1. Overview of NVMe over fabric devices

Non-volatile Memory Express™ (NVMe™) is an interface that allows host software utility to communicate with solid state drives.

Use the following types of fabric transport to configure NVMe over fabric devices:

NVMe over Remote Direct Memory Access (NVMe/RDMA)
For information about how to configure NVMe™/RDMA, see Configuring NVMe over fabrics using NVMe/RDMA.
NVMe over Fibre Channel (NVMe/FC)
For information about how to configure NVMe™/FC, see Configuring NVMe over fabrics using NVMe/FC.
NVMe over TCP (NVMe/TCP)
For information about how to configure NVMe/FC, see Configuring NVMe over fabrics using NVMe/TCP.

When using NVMe over fabrics, the solid-state drive does not have to be local to your system; it can be configured remotely through a NVMe over fabrics devices.

13.2. Configuring an NVMe/TCP host

Use the Non-volatile Memory Express™ (NVMe™) management command line interface (nvme-cli) tool to configure an NVMe/TCP host.

Procedure

  1. Install the nvme-cli tool:

    # dnf install nvme-cli

    This tool creates the hostnqn file in the /etc/nvme/ directory, which identifies the NVMe host.

  2. Find the nvme hostid and hostnqn:

    # cat /etc/nvme/hostnqn
    nqn.2014-08.org.nvmexpress:uuid:8ae2b12c-3d28-4458-83e3-658e571ed4b8
    
    # cat /etc/nvme/hostid
    09e2ce17-ccc9-412d-8dcf-2b0a1d581ee3

    Use the hostid and hostnqn values to configure the NVMe/TCP controller.

  3. Check the status of the controller:

    # nmcli device show ens6
    GENERAL.DEVICE:                         ens6
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         52:57:02:12:02:02
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          30 (disconnected)
    GENERAL.CONNECTION:                     --
    GENERAL.CON-PATH:                       --
    WIRED-PROPERTIES.CARRIER:               on
  4. Configure the host network for a newly installed Ethernet controller with a static IP address:

    # nmcli connection add con-name ens6 ifname ens6 type ethernet ip4 192.168.101.154/24 gw4 192.168.101.1

    Here, replace 192.168.101.154 with the host IP address.

    # nmcli connection mod ens6 ipv4.method manual
    # nmcli connection up ens6

    Since a new network is created to connect the NVMe/TCP host to the NVMe/TCP controller, execute this step on the controller too.

Verification

  • Verify if the newly created host network works correctly:

    # nmcli device show ens6
    GENERAL.DEVICE:                         ens6
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         52:57:02:12:02:02
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     ens6
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/5
    WIRED-PROPERTIES.CARRIER:               on
    IP4.ADDRESS[1]:                         192.168.101.154/24
    IP4.GATEWAY:                            192.168.101.1
    IP4.ROUTE[1]:                           dst = 192.168.101.0/24, nh = 0.0.0.0, mt = 101
    IP4.ROUTE[2]:                           dst = 192.168.1.1/32, nh = 0.0.0.0, mt = 101
    IP4.ROUTE[3]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 101
    IP6.ADDRESS[1]:                         fe80::27ce:dde1:620:996c/64
    IP6.GATEWAY:                            --
    IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 101

Additional resources

  • The nvme(1) man page

13.3. Connecting the NVMe/TCP host to the NVMe/TCP controller

Connect the NVMe™ over TCP (NVMe/TCP) host to the NVMe/TCP controller system to verify that the NVMe/TCP host can now access the namespace.

Note

The NVMe/TCP controller (nvmet_tcp) module is not supported.

Prerequisites

  • You have configured an NVMe/TCP host. For more information, see Configuring an NVMe/TCP host.
  • You have configured an NVMe/TCP controller using external storage software and the network is configured on the controller. In this procedure, 192.168.101.55 is the IP address of NVMe/TCP controller.

Procedure

  1. Load the nvme_tcp module if not already:

    # modprobe nvme_tcp
  2. Discover the available subsystems on the NVMe controller:

    # nvme discover --transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009
    
    Discovery Log Number of Records 2, Generation counter 7
    =====Discovery Log Entry 0======
    trtype:  tcp
    adrfam:  ipv4
    subtype: current discovery subsystem
    treq:	not specified, sq flow control disable supported
    portid:  2
    trsvcid: 8009
    subnqn:  nqn.2014-08.org.nvmexpress.discovery
    traddr:  192.168.101.55
    eflags:  not specified
    sectype: none
    =====Discovery Log Entry 1======
    trtype:  tcp
    adrfam:  ipv4
    subtype: nvme subsystem
    treq:	not specified, sq flow control disable supported
    portid:  2
    trsvcid: 8009
    subnqn:  nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
    traddr:  192.168.101.55
    eflags:  not specified
    sectype: none

    Here, 192.168.101.55 is the NVMe/TCP controller IP address and 192.168.101.154 is the NVMe/TCP host IP address.

  3. Configure the /etc/nvme/discovery.conf file to add the parameters used in the nvme discover command :

    # echo "--transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009" >> /etc/nvme/discovery.conf
  4. Connect the NVMe/TCP host to the controller system:

    # nvme connect-all

Verification

  • Verify that the NVMe/TCP host can access the namespace:

    # nvme list-subsys
    
    nvme-subsys3 - NQN=nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
    \
     +- nvme3 tcp traddr=192.168.101.55,trsvcid=8009,host_traddr=192.168.101.154 live optimized
    
    # nvme list
    Node              	Generic           	SN               	Model                                	Namespace Usage                  	Format       	FW Rev
    --------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
    /dev/nvme3n1      	/dev/ng3n1        	d93a63d394d043ab4b74 Linux                                    1          21.47  GB /  21.47  GB    512   B +  0 B   5.18.5-2

Additional resources

  • The nvme(1) man page
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.