Chapter 4. Enabling multipathing on NVMe devices

Procedure

1. Check if the native NVMe multipathing is enabled:

   Copy to Clipboard Toggle word wrap

   # cat /sys/module/nvme_core/parameters/multipath

   The command displays one of the following:

   N

   Native NVMe multipathing is disabled.

   Y

   Native NVMe multipathing is enabled.

2. If the native NVMe multipathing is enabled, disable it by using one of the following methods:

   • Using a kernel option:

     1. Add the nvme_core.multipath=N option to the command line:

        Copy to Clipboard Toggle word wrap

        # grubby --update-kernel=ALL --args="nvme_core.multipath=N"

     2. On the 64-bit IBM Z architecture, update the boot menu:

        Copy to Clipboard Toggle word wrap

        # zipl

     3. Reboot the system.

   • Using a kernel module configuration file:

     1. Create the /etc/modprobe.d/nvme_core.conf configuration file with the following content:

        Copy to Clipboard Toggle word wrap

        options nvme_core multipath=N

     2. Back up the initramfs file:

        Copy to Clipboard Toggle word wrap

        # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m%d-%H%M%S).img

     3. Rebuild the initramfs:

        Copy to Clipboard Toggle word wrap

        # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img
        # dracut --force --verbose

     4. Reboot the system.

3. Enable DM Multipath:

   Copy to Clipboard Toggle word wrap

   # systemctl enable --now multipathd.service

4. Distribute I/O on all available paths. Add the following content in the /etc/multipath.conf file:

   Copy to Clipboard Toggle word wrap

   devices {
           device {
                   vendor "NVME"
                   product ".*"
                   path_grouping_policy    group_by_prio
           }
   }

   Note

   The /sys/class/nvme-subsystem/nvme-subsys0/iopolicy configuration file has no effect on the I/O distribution when DM Multipath manages the NVMe devices.

5. Reload the multipathd service to apply the configuration changes:

   Copy to Clipboard Toggle word wrap

   # multipath -r

Procedure

1. Check if native NVMe multipathing is enabled in the kernel:

   Copy to Clipboard Toggle word wrap

   # cat /sys/module/nvme_core/parameters/multipath

   The command displays one of the following:

   N

   Native NVMe multipathing is disabled.

   Y

   Native NVMe multipathing is enabled.

2. If native NVMe multipathing is disabled, enable it by using one of the following methods:

   • Using a kernel option:

     1. Remove the nvme_core.multipath=N option from the kernel command line:

        Copy to Clipboard Toggle word wrap

        # grubby --update-kernel=ALL --remove-args="nvme_core.multipath=N"

     2. On the 64-bit IBM Z architecture, update the boot menu:

        Copy to Clipboard Toggle word wrap

        # zipl

     3. Reboot the system.

   • Using a kernel module configuration file:

     1. Remove the /etc/modprobe.d/nvme_core.conf configuration file:

        Copy to Clipboard Toggle word wrap

        # rm /etc/modprobe.d/nvme_core.conf

     2. Back up the initramfs file:

        Copy to Clipboard Toggle word wrap

        # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

     3. Rebuild the initramfs:

        Copy to Clipboard Toggle word wrap

        # dracut --force --verbose

     4. Reboot the system.

3. Optional: On the running system, change the I/O policy on NVMe devices to distribute the I/O on all available paths:

   Copy to Clipboard Toggle word wrap

   # echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys0/iopolicy

4. Optional: Set the I/O policy persistently using udev rules. Create the /etc/udev/rules.d/71-nvme-io-policy.rules file with the following content:

   Copy to Clipboard Toggle word wrap

   ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="round-robin"

Verification

1. Verify if your system recognizes the NVMe devices. The following example assumes you have a connected NVMe over fabrics storage subsystem with two NVMe namespaces:

   Copy to Clipboard Toggle word wrap

   # nvme list
   
   Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
   ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
   /dev/nvme0n1     a34c4f3a0d6f5cec     Linux                                    1         250.06  GB / 250.06  GB    512   B +  0 B   4.18.0-2
   /dev/nvme0n2     a34c4f3a0d6f5cec     Linux                                    2         250.06  GB / 250.06  GB    512   B +  0 B   4.18.0-2

2. List all connected NVMe subsystems:

   Copy to Clipboard Toggle word wrap

   # nvme list-subsys
   
   nvme-subsys0 - NQN=testnqn
   \
    +- nvme0 fc traddr=nn-0x20000090fadd597a:pn-0x10000090fadd597a host_traddr=nn-0x20000090fac7e1dd:pn-0x10000090fac7e1dd live
    +- nvme1 fc traddr=nn-0x20000090fadd5979:pn-0x10000090fadd5979 host_traddr=nn-0x20000090fac7e1dd:pn-0x10000090fac7e1dd live
    +- nvme2 fc traddr=nn-0x20000090fadd5979:pn-0x10000090fadd5979 host_traddr=nn-0x20000090fac7e1de:pn-0x10000090fac7e1de live
    +- nvme3 fc traddr=nn-0x20000090fadd597a:pn-0x10000090fadd597a host_traddr=nn-0x20000090fac7e1de:pn-0x10000090fac7e1de live

   Check the active transport type. For example, nvme0 fc indicates that the device is connected over the Fibre Channel transport, and nvme tcp indicates that the device is connected over TCP.

3. If you edited the kernel options, verify if native NVMe multipathing is enabled on the kernel command line:

   Copy to Clipboard Toggle word wrap

   # cat /proc/cmdline
   
   BOOT_IMAGE=[...] nvme_core.multipath=Y

4. If you changed the I/O policy, verify if round-robin is the active I/O policy on NVMe devices:

   Copy to Clipboard Toggle word wrap

   # cat /sys/class/nvme-subsystem/nvme-subsys0/iopolicy
   
   round-robin