Chapter 12. Configuring NVMe over fabrics using NVMe/TCP

Procedure

1. Install the nvme-cli tool:

   Copy to Clipboard Toggle word wrap

   # dnf install nvme-cli

   This tool creates the hostnqn file in the /etc/nvme/ directory, which identifies the NVMe host.

2. Find the nvme hostid and hostnqn:

   Copy to Clipboard Toggle word wrap

   # cat /etc/nvme/hostnqn
   nqn.2014-08.org.nvmexpress:uuid:8ae2b12c-3d28-4458-83e3-658e571ed4b8
   
   # cat /etc/nvme/hostid
   09e2ce17-ccc9-412d-8dcf-2b0a1d581ee3

   Use the hostid and hostnqn values to configure the NVMe/TCP controller.

3. Check the status of the controller:

   Copy to Clipboard Toggle word wrap

   # nmcli device show ens6
   GENERAL.DEVICE:                         ens6
   GENERAL.TYPE:                           ethernet
   GENERAL.HWADDR:                         52:57:02:12:02:02
   GENERAL.MTU:                            1500
   GENERAL.STATE:                          30 (disconnected)
   GENERAL.CONNECTION:                     --
   GENERAL.CON-PATH:                       --
   WIRED-PROPERTIES.CARRIER:               on

4. Configure the host network for a newly installed Ethernet controller with a static IP address:

   Copy to Clipboard Toggle word wrap

   # nmcli connection add con-name ens6 ifname ens6 type ethernet ip4 192.168.101.154/24 gw4 192.168.101.1

   Here, replace 192.168.101.154 with the host IP address.

   Copy to Clipboard Toggle word wrap

   # nmcli connection mod ens6 ipv4.method manual
   # nmcli connection up ens6

   Since a new network is created to connect the NVMe/TCP host to the NVMe/TCP controller, execute this step on the controller too.

Procedure

1. Load the nvme-tcp module if not already:

   Copy to Clipboard Toggle word wrap

   # modprobe nvme-tcp

2. Discover the available subsystems on the NVMe controller:

   Copy to Clipboard Toggle word wrap

   # nvme discover --transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009
   
   Discovery Log Number of Records 2, Generation counter 7
   =====Discovery Log Entry 0======
   trtype:  tcp
   adrfam:  ipv4
   subtype: current discovery subsystem
   treq:	not specified, sq flow control disable supported
   portid:  2
   trsvcid: 8009
   subnqn:  nqn.2014-08.org.nvmexpress.discovery
   traddr:  192.168.101.55
   eflags:  not specified
   sectype: none
   =====Discovery Log Entry 1======
   trtype:  tcp
   adrfam:  ipv4
   subtype: nvme subsystem
   treq:	not specified, sq flow control disable supported
   portid:  2
   trsvcid: 8009
   subnqn:  nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
   traddr:  192.168.101.55
   eflags:  not specified
   sectype: none

   Here, 192.168.101.55 is the NVMe/TCP controller IP address and 192.168.101.154 is the NVMe/TCP host IP address.

3. Configure the /etc/nvme/discovery.conf file to add the parameters used in the nvme discover command :

   Copy to Clipboard Toggle word wrap

   # echo "--transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009" >> /etc/nvme/discovery.conf

4. Connect the NVMe/TCP host to the controller system:

   Copy to Clipboard Toggle word wrap

   # nvme connect-all

Procedure

1. Generate an authentication secret:

   1. For the host:

      Copy to Clipboard Toggle word wrap

      # hostkey=$(nvme gen-dhchap-key -n ${HOSTNQN})

   2. For the subsystem:

      Copy to Clipboard Toggle word wrap

      # ctrlkey= $(nvme gen-dhchap-key -n ${SUBSYSTEM})

2. Configure the host for authentication:

   Copy to Clipboard Toggle word wrap

   # nvme connect -t tcp -n ${SUBSYSTEM} -a ${TRADDR} -s 4420 --dhchap-secret=${hostkey} --dhchap-ctrl-secret=${ctrlkey}

   This provides the authentication secrets to the nvme-connect utility so that it can authenticate and establish a connection to the target.

   • Optional: To enable automated logins, set up persistent NVMe fabrics configuration. To do so, add the --dhchap-secret and --dhchap-ctrl-secret parameters to /etc/nvme/discovery.conf or /etc/nvme/config.json.

Procedure

1. Configure Pre-Shared-Key Keyring.

   1. Identify Host NQN:

      Copy to Clipboard Toggle word wrap

      # HOSTNQN=$(nvme show-hostnqn)

   2. Generate and copy a newly configured PSK:

      Copy to Clipboard Toggle word wrap

      # PSK=$(nvme gen-tls-key)

      Copy to Clipboard Toggle word wrap

      # echo $PSK

   3. Configure Pre-Shared-Key Keyring:

      Copy to Clipboard Toggle word wrap

      # nvme check-tls-key --insert --hostnqn=${HOSTNQN} --subsysnqn=${SUBSYSTEM} --keydata=${PSK} --identity=1

2. Configure the tlshd service.

   1. Add the keyring name to the /etc/tlshd.conf configuration file:

      Copy to Clipboard Toggle word wrap

      ...
      [authenticate]
      keyring=.nvme
      ...

   2. Restart the tlshd service:

      Copy to Clipboard Toggle word wrap

      # systemctl restart tlshd

3. Enable TLS for NVMe fabrics connections:

   Copy to Clipboard Toggle word wrap

   # nvme discover -t tcp --tls -a ${TRADDR} -s 4420

   Copy to Clipboard Toggle word wrap

   # nvme connect -t tcp --tls -a ${TRADDR} -s 4420 -n ${SUBSYSTEM}