Managing storage devices

Procedure

1. Start the interactive parted shell:

   # parted block-device

   Copy to Clipboard Toggle word wrap

2. Determine if there already is a partition table on the device:

   (parted) print

   Copy to Clipboard Toggle word wrap

   If the device already contains partitions, they will be deleted in the following steps.

3. Create the new partition table:

   (parted) mklabel table-type

   Copy to Clipboard Toggle word wrap

   • Replace table-type with with the intended partition table type:

     • msdos for MBR
     • gpt for GPT

   Example 4.1. Creating a GUID Partition Table (GPT) table

   To create a GPT table on the disk, use:

   (parted) mklabel gpt

   Copy to Clipboard Toggle word wrap

   The changes start applying after you enter this command.

4. View the partition table to confirm that it is created:

   (parted) print

   Copy to Clipboard Toggle word wrap

5. Exit the parted shell:

   (parted) quit

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the parted utility. For example, the following output lists the device /dev/sda:

   # parted /dev/sda

   Copy to Clipboard Toggle word wrap

2. View the partition table:

   (parted) print
   
   Model: ATA SAMSUNG MZNLN256 (scsi)
   Disk /dev/sda: 256GB
   Sector size (logical/physical): 512B/512B
   Partition Table: msdos
   Disk Flags:
   
   Number  Start   End     Size    Type      File system  Flags
    1      1049kB  269MB   268MB   primary   xfs          boot
    2      269MB   34.6GB  34.4GB  primary
    3      34.6GB  45.4GB  10.7GB  primary
    4      45.4GB  256GB   211GB   extended
    5      45.4GB  256GB   211GB   logical

   Copy to Clipboard Toggle word wrap

3. Optional: Switch to the device you want to examine next:

   (parted) select block-device

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the parted utility:

   # parted block-device

   Copy to Clipboard Toggle word wrap

2. View the current partition table to determine if there is enough free space:

   (parted) print

   Copy to Clipboard Toggle word wrap
   • Resize the partition in case there is not enough free space.

   • From the partition table, determine:

     • The start and end points of the new partition.
     • On MBR, what partition type it should be.

3. Create the new partition:

   (parted) mkpart part-type name fs-type start end

   Copy to Clipboard Toggle word wrap
   • Replace part-type with with primary, logical, or extended. This applies only to the MBR partition table.
   • Replace name with an arbitrary partition name. This is required for GPT partition tables.
   • Replace fs-type with xfs, ext2, ext3, ext4, fat16, fat32, hfs, hfs+, linux-swap, ntfs, or reiserfs. The fs-type parameter is optional. Note that the parted utility does not create the file system on the partition.
   • Replace start and end with the sizes that determine the starting and ending points of the partition, counting from the beginning of the disk. You can use size suffixes, such as 512MiB, 20GiB, or 1.5TiB. The default size is in megabytes.

   Example 4.2. Creating a small primary partition

   To create a primary partition from 1024MiB until 2048MiB on an MBR table, use:

   (parted) mkpart primary 1024MiB 2048MiB

   Copy to Clipboard Toggle word wrap

   The changes start applying after you enter the command.

4. View the partition table to confirm that the created partition is in the partition table with the correct partition type, file system type, and size:

   (parted) print

   Copy to Clipboard Toggle word wrap

5. Exit the parted shell:

   (parted) quit

   Copy to Clipboard Toggle word wrap

6. Register the new device node:

   # udevadm settle

   Copy to Clipboard Toggle word wrap

7. Verify that the kernel recognizes the new partition:

   # cat /proc/partitions

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the interactive fdisk shell:

   # fdisk block-device

   Copy to Clipboard Toggle word wrap

2. View the current partition table to determine the minor partition number:

   Command (m for help): print

   Copy to Clipboard Toggle word wrap

   You can see the current partition type in the Type column and its corresponding type ID in the Id column.

3. Enter the partition type command and select a partition using its minor number:

   Command (m for help): type
   Partition number (1,2,3 default 3): 2

   Copy to Clipboard Toggle word wrap

4. Optional: View the list in hexadecimal codes:

   Hex code (type L to list all codes): L

   Copy to Clipboard Toggle word wrap

5. Set the partition type:

   Hex code (type L to list all codes): 8e

   Copy to Clipboard Toggle word wrap

6. Write your changes and exit the fdisk shell:

   Command (m for help): write
   The partition table has been altered.
   Syncing disks.

   Copy to Clipboard Toggle word wrap

7. Verify your changes:

   # fdisk --list block-device

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the parted utility:

   # parted block-device

   Copy to Clipboard Toggle word wrap

2. View the current partition table:

   (parted) print

   Copy to Clipboard Toggle word wrap

   From the partition table, determine:

   • The minor number of the partition.
   • The location of the existing partition and its new ending point after resizing.

3. Resize the partition:

   (parted) resizepart 1 2GiB

   Copy to Clipboard Toggle word wrap
   • Replace 1 with the minor number of the partition that you are resizing.
   • Replace 2 with the size that determines the new ending point of the resized partition, counting from the beginning of the disk. You can use size suffixes, such as 512MiB, 20GiB, or 1.5TiB. The default size is in megabytes.

4. View the partition table to confirm that the resized partition is in the partition table with the correct size:

   (parted) print

   Copy to Clipboard Toggle word wrap

5. Exit the parted shell:

   (parted) quit

   Copy to Clipboard Toggle word wrap

6. Verify that the kernel registers the new partition:

   # cat /proc/partitions

   Copy to Clipboard Toggle word wrap
7. Optional: If you extended the partition, extend the file system on it as well.

Procedure

1. Start the interactive parted shell:

   # parted block-device

   Copy to Clipboard Toggle word wrap
   • Replace block-device with the path to the device where you want to remove a partition: for example, /dev/sda.

2. View the current partition table to determine the minor number of the partition to remove:

   (parted) print

   Copy to Clipboard Toggle word wrap

3. Remove the partition:

   (parted) rm minor-number

   Copy to Clipboard Toggle word wrap
   • Replace minor-number with the minor number of the partition you want to remove.

   The changes start applying as soon as you enter this command.

4. Verify that you have removed the partition from the partition table:

   (parted) print

   Copy to Clipboard Toggle word wrap

5. Exit the parted shell:

   (parted) quit

   Copy to Clipboard Toggle word wrap

6. Verify that the kernel registers that the partition is removed:

   # cat /proc/partitions

   Copy to Clipboard Toggle word wrap
7. Remove the partition from the /etc/fstab file, if it is present. Find the line that declares the removed partition, and remove it from the file.

8. Regenerate mount units so that your system registers the new /etc/fstab configuration:

   # systemctl daemon-reload

   Copy to Clipboard Toggle word wrap

9. If you have deleted a swap partition or removed pieces of LVM, remove all references to the partition from the kernel command line:

   1. List active kernel options and see if any option references the removed partition:

      # grubby --info=ALL

      Copy to Clipboard Toggle word wrap

   2. Remove the kernel options that reference the removed partition:

      # grubby --update-kernel=ALL --remove-args="option"

      Copy to Clipboard Toggle word wrap

10. To register the changes in the early boot system, rebuild the initramfs file system:

    # dracut --force --verbose

    Copy to Clipboard Toggle word wrap

Procedure

1. Install the targetcli tool:

   # dnf install targetcli

   Copy to Clipboard Toggle word wrap

2. Start the target service:

   # systemctl start target

   Copy to Clipboard Toggle word wrap

3. Configure target to start at boot time:

   # systemctl enable target

   Copy to Clipboard Toggle word wrap

4. Open port 3260 in the firewall and reload the firewall configuration:

   # firewall-cmd --permanent --add-port=3260/tcp
   Success

   Copy to Clipboard Toggle word wrap

   # firewall-cmd --reload
   Success

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the iSCSI directory. You can also use the cd command to navigate to the iSCSI directory.

   /> iscsi/

   Copy to Clipboard Toggle word wrap

2. Use one of the following options to create an iSCSI target:

   1. Creating an iSCSI target using a default target name:

      /iscsi> create
      
      Created target
      iqn.2003-01.org.linux-iscsi.hostname.x8664:sn.78b473f296ff
      Created TPG1

      Copy to Clipboard Toggle word wrap

   2. Creating an iSCSI target using a specific name:

      /iscsi> create iqn.2006-04.com.example:444
      
      Created target iqn.2006-04.com.example:444
      Created TPG1
      Here iqn.2006-04.com.example:444 is target_iqn_name

      Copy to Clipboard Toggle word wrap

      Replace iqn.2006-04.com.example:444 with the specific target name.

3. Verify the newly created target:

   /iscsi> ls
   
   o- iscsi.......................................[1 Target]
       o- iqn.2006-04.com.example:444................[1 TPG]
           o- tpg1...........................[enabled, auth]
              o- acls...............................[0 ACL]
               o- luns...............................[0 LUN]
              o- portals.........................[0 Portal]

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the fileio/ from the backstores/ directory:

   /> backstores/fileio

   Copy to Clipboard Toggle word wrap

2. Create a fileio storage object:

   /backstores/fileio> create file1 /tmp/disk1.img 200M write_back=false
   
   Created fileio file1 with size 209715200

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the block/ from the backstores/ directory:

   /> backstores/block/

   Copy to Clipboard Toggle word wrap

2. Create a block backstore:

   /backstores/block> create name=block_backend dev=/dev/sdb
   
   Generating a wwn serial.
   Created block storage object block_backend using /dev/sdb.

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the pscsi/ from the backstores/ directory:

   /> backstores/pscsi/

   Copy to Clipboard Toggle word wrap

2. Create a pscsi backstore for a physical SCSI device, a TYPE_ROM device using /dev/sr0 in this example:

   /backstores/pscsi> create name=pscsi_backend dev=/dev/sr0
   
   Generating a wwn serial.
   Created pscsi storage object pscsi_backend using /dev/sr0

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the ramdisk/ from the backstores/ directory:

   /> backstores/ramdisk/

   Copy to Clipboard Toggle word wrap

2. Create a 1GB RAM disk backstore:

   /backstores/ramdisk> create name=rd_backend size=1GB
   
   Generating a wwn serial.
   Created rd_mcp ramdisk rd_backend with size 1GB.

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the TPG directory:

   /iscsi> iqn.2006-04.com.example:444/tpg1/

   Copy to Clipboard Toggle word wrap

2. Use one of the following options to create an iSCSI portal:

   1. Creating a default portal uses the default iSCSI port 3260 and allows the target to listen to all IP addresses on that port:

      /iscsi/iqn.20...mple:444/tpg1> portals/ create
      
      Using default IP port 3260
      Binding to INADDR_Any (0.0.0.0)
      Created network portal 0.0.0.0:3260

      Copy to Clipboard Toggle word wrap

   2. Creating a portal using a specific IP address:

      /iscsi/iqn.20...mple:444/tpg1> portals/ create 192.168.122.137
      
      Using default IP port 3260
      Created network portal 192.168.122.137:3260

      Copy to Clipboard Toggle word wrap

Procedure

1. Create LUNs of already created storage objects:

   /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/ramdisk/rd_backend
   Created LUN 0.
   
   /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/block/block_backend
   Created LUN 1.
   
   /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/fileio/file1
   Created LUN 2.

   Copy to Clipboard Toggle word wrap

2. Verify the created LUNs:

   /iscsi/iqn.20...mple:444/tpg1> ls
   
   o- tpg.................................. [enabled, auth]
       o- acls ......................................[0 ACL]
       o- luns .....................................[3 LUNs]
       |  o- lun0.........................[ramdisk/ramdisk1]
       |  o- lun1.................[block/block1 (/dev/vdb1)]
       |  o- lun2...................[fileio/file1 (/foo.img)]
       o- portals ................................[1 Portal]
           o- 192.168.122.137:3260......................[OK]

   Copy to Clipboard Toggle word wrap

   Default LUN name starts at 0.

   Important

   By default, LUNs are created with read-write permissions. If a new LUN is added after ACLs are created, LUN automatically maps to all available ACLs and can cause a security risk. To create a LUN with read-only permissions, see Creating a read-only iSCSI LUN.

3. Configure ACLs. For more information, see Creating an iSCSI ACL.

Procedure

1. Set read-only permissions:

   /> set global auto_add_mapped_luns=false
   
   Parameter auto_add_mapped_luns is now 'false'.

   Copy to Clipboard Toggle word wrap

   This prevents the auto mapping of LUNs to existing ACLs allowing the manual mapping of LUNs.

2. Navigate to the initiator_iqn_name directory:

   /> iscsi/target_iqn_name/tpg1/acls/initiator_iqn_name/

   Copy to Clipboard Toggle word wrap

3. Create the LUN:

   /iscsi/target_iqn_name/tpg1/acls/initiator_iqn_name> create mapped_lun=next_sequential_LUN_number tpg_lun_or_backstore=backstore write_protect=1

   Copy to Clipboard Toggle word wrap

   Example:

   /iscsi/target_iqn_name/tpg1/acls/2006-04.com.example:888> create mapped_lun=1 tpg_lun_or_backstore=/backstores/block/block2 write_protect=1
   
   Created LUN 1.
   Created Mapped LUN 1.

   Copy to Clipboard Toggle word wrap

4. Verify the created LUN:

   /iscsi/target_iqn_name/tpg1/acls/2006-04.com.example:888> ls
    o- 2006-04.com.example:888 .. [Mapped LUNs: 2]
    | o- mapped_lun0 .............. [lun0 block/disk1 (rw)]
    | o- mapped_lun1 .............. [lun1 block/disk2 (ro)]

   Copy to Clipboard Toggle word wrap

   The mapped_lun1 line now has (ro) at the end (unlike mapped_lun0’s (rw)) stating that it is read-only.

5. Configure ACLs. For more information, see Creating an iSCSI ACL.

Procedure

1. Optional: To disable auto mapping of LUNs to ACLs see Creating a read-only iSCSI LUN.

2. Navigate to the acls directory:

   /> iscsi/target_iqn_name/tpg_name/acls/

   Copy to Clipboard Toggle word wrap

3. Use one of the following options to create an ACL:

   • Use the initiator_iqn_name from the /etc/iscsi/initiatorname.iscsi file on the initiator:

     iscsi/target_iqn_name/tpg_name/acls> create initiator_iqn_name
     
     Created Node ACL for initiator_iqn_name
     Created mapped LUN 2.
     Created mapped LUN 1.
     Created mapped LUN 0.

     Copy to Clipboard Toggle word wrap

   • Use a custom_name and update the initiator to match it:

     iscsi/target_iqn_name/tpg_name/acls> create custom_name
     
     Created Node ACL for custom_name
     Created mapped LUN 2.
     Created mapped LUN 1.
     Created mapped LUN 0.

     Copy to Clipboard Toggle word wrap

     For information about updating the initiator name, see Creating an iSCSI intiator.

Procedure

1. Set attribute authentication:

   /iscsi/iqn.20...mple:444/tpg1> set attribute authentication=1
   
   Parameter authentication is now '1'.

   Copy to Clipboard Toggle word wrap

2. Set userid and password:

   /tpg1> set auth userid=redhat
   Parameter userid is now 'redhat'.
   
   /iscsi/iqn.20...689dcbb3/tpg1> set auth password=redhat_passwd
   Parameter password is now 'redhat_passwd'.

   Copy to Clipboard Toggle word wrap

3. Navigate to the acls directory:

   /> iscsi/target_iqn_name/tpg1/acls/initiator_iqn_name/

   Copy to Clipboard Toggle word wrap

4. Set attribute authentication:

   /iscsi/iqn.20...:605fcc6a48be> set attribute authentication=1
   Parameter authentication is now '1'.

   Copy to Clipboard Toggle word wrap

5. Set userid and password:

   /iscsi/iqn.20...:605fcc6a48be> set auth userid=redhat
   Parameter userid is now 'redhat'.
   
   /iscsi/iqn.20...:605fcc6a48be> set auth password=redhat_passwd
   Parameter password is now 'redhat_passwd'.

   Copy to Clipboard Toggle word wrap

Procedure

1. Log off from the target:

   # iscsiadm -m node -T iqn.2006-04.com.example:444 -u

   Copy to Clipboard Toggle word wrap

   For more information about how to log in to the target, see Creating an iSCSI initiator.

2. Remove the entire target, including all ACLs, LUNs, and portals:

   /> iscsi/ delete iqn.2006-04.com.example:444

   Copy to Clipboard Toggle word wrap

   Replace iqn.2006-04.com.example:444 with the target_iqn_name.

   • To remove an iSCSI backstore:

     /> backstores/backstore-type/ delete block_backend

     Copy to Clipboard Toggle word wrap

     Replace backstore-type with either fileio, block, pscsi, or ramdisk.

     Replace block_backend with the backstore-name you want to delete.

   • To remove parts of an iSCSI target, such as an ACL:

     /> /iscsi/iqn-name/tpg/acls/ delete iqn.2006-04.com.example:444

     Copy to Clipboard Toggle word wrap

Procedure

1. Install iscsi-initiator-utils on client machine:

   # dnf install iscsi-initiator-utils

   Copy to Clipboard Toggle word wrap

2. Start the iscsid service:

   # systemctl start iscsid

   Copy to Clipboard Toggle word wrap

3. Check the initiator name:

   # cat /etc/iscsi/initiatorname.iscsi
   
   InitiatorName=iqn.2006-04.com.example:888

   Copy to Clipboard Toggle word wrap

4. If the ACL was given a custom name in Creating an iSCI ACL, update the initiator name to match the ACL:

   1. Open the /etc/iscsi/initiatorname.iscsi file and modify the initiator name:

      # vi /etc/iscsi/initiatorname.iscsi
      
      InitiatorName=custom-name

      Copy to Clipboard Toggle word wrap

   2. Restart the iscsid service:

      # systemctl restart iscsid

      Copy to Clipboard Toggle word wrap

5. Discover the target and log in to the target with the displayed target IQN:

   # iscsiadm -m discovery -t st -p 10.64.24.179
       10.64.24.179:3260,1 iqn.2006-04.com.example:444
   
   # iscsiadm -m node -T iqn.2006-04.com.example:444 -l
       Logging in to [iface: default, target: iqn.2006-04.com.example:444, portal: 10.64.24.179,3260] (multiple)
       Login to [iface: default, target: iqn.2006-04.com.example:444, portal: 10.64.24.179,3260] successful.

   Copy to Clipboard Toggle word wrap

   Replace 10.64.24.179 with the target-ip-address.

   You can use this procedure for any number of initiators connected to the same target if their respective initiator names are added to the ACL as described in the Creating an iSCSI ACL.

6. Find the iSCSI disk name and create a file system on this iSCSI disk:

   # grep "Attached SCSI" /var/log/messages
   
   # mkfs.ext4 /dev/disk_name

   Copy to Clipboard Toggle word wrap

   Replace disk_name with the iSCSI disk name displayed in the /var/log/messages file.

7. Mount the file system:

   # mkdir /mount/point
   
   # mount /dev/disk_name /mount/point

   Copy to Clipboard Toggle word wrap

   Replace /mount/point with the mount point of the partition.

8. Edit the /etc/fstab file to mount the file system automatically when the system boots:

   # vi /etc/fstab
   
   /dev/disk_name /mount/point ext4 _netdev 0 0

   Copy to Clipboard Toggle word wrap

   Replace disk_name with the iSCSI disk name and /mount/point with the mount point of the partition.

Procedure

1. Enable CHAP authentication in the iscsid.conf file:

   # vi /etc/iscsi/iscsid.conf
   
   node.session.auth.authmethod = CHAP

   Copy to Clipboard Toggle word wrap

   By default, the node.session.auth.authmethod is set to None

2. Add target username and password in the iscsid.conf file:

   node.session.auth.username = redhat
   node.session.auth.password = redhat_passwd

   Copy to Clipboard Toggle word wrap

3. Restart the iscsid service:

   # systemctl restart iscsid

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the iscsi-initiator-utils on client machine:

   # dnf install iscsi-initiator-utils

   Copy to Clipboard Toggle word wrap

2. Find information about the running sessions:

   # iscsiadm -m session -P 3

   Copy to Clipboard Toggle word wrap

   This command displays the session or device state, session ID (sid), some negotiated parameters, and the SCSI devices accessible through the session.

   • For shorter output, for example, to display only the sid-to-node mapping, run:

     # iscsiadm -m session -P 0
             or
     # iscsiadm -m session
     
     tcp [2] 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
     tcp [3] 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311

     Copy to Clipboard Toggle word wrap

     These commands print the list of running sessions in the following format: driver [sid] target_ip:port,target_portal_group_tag proper_target_name.

Procedure

1. Determine which devices are paths for a multipath logical unit:

   # multipath -ll

   Copy to Clipboard Toggle word wrap

2. Re-scan Fibre Channel logical units on a system that uses multipathing:

   $ echo 1 > /sys/block/<device_ID>/device/rescan

   Copy to Clipboard Toggle word wrap

   Replace <device_ID> with the ID of your device, for example sda.

Procedure

1. Create the nvmet-rdma subsystem:

   # modprobe nvmet-rdma
   
   # mkdir /sys/kernel/config/nvmet/subsystems/testnqn
   
   # cd /sys/kernel/config/nvmet/subsystems/testnqn

   Copy to Clipboard Toggle word wrap

   Replace testnqn with the subsystem name.

2. Allow any host to connect to this controller:

   # echo 1 > attr_allow_any_host

   Copy to Clipboard Toggle word wrap

3. Configure a namespace:

   # mkdir namespaces/10
   
   # cd namespaces/10

   Copy to Clipboard Toggle word wrap

   Replace 10 with the namespace number

4. Set a path to the NVMe device:

   # echo -n /dev/nvme0n1 > device_path

   Copy to Clipboard Toggle word wrap

5. Enable the namespace:

   # echo 1 > enable

   Copy to Clipboard Toggle word wrap

6. Create a directory with an NVMe port:

   # mkdir /sys/kernel/config/nvmet/ports/1
   
   # cd /sys/kernel/config/nvmet/ports/1

   Copy to Clipboard Toggle word wrap

7. Display the IP address of mlx5_ib0:

   # ip addr show mlx5_ib0
   
   8: mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP group default qlen 256
       link/infiniband 00:00:06:2f:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:e7:0f:f6 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
       inet 172.31.0.202/24 brd 172.31.0.255 scope global noprefixroute mlx5_ib0
          valid_lft forever preferred_lft forever
       inet6 fe80::e61d:2d03:e7:ff6/64 scope link noprefixroute
          valid_lft forever preferred_lft forever

   Copy to Clipboard Toggle word wrap

8. Set the transport address for the controller:

   # echo -n 172.31.0.202 > addr_traddr

   Copy to Clipboard Toggle word wrap

9. Set RDMA as the transport type:

   # echo rdma > addr_trtype
   
   # echo 4420 > addr_trsvcid

   Copy to Clipboard Toggle word wrap

10. Set the address family for the port:

    # echo ipv4 > addr_adrfam

    Copy to Clipboard Toggle word wrap

11. Create a soft link:

    # ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/testnqn

    Copy to Clipboard Toggle word wrap

Procedure

1. Install the nvmetcli package:

   # dnf install nvmetcli

   Copy to Clipboard Toggle word wrap

2. Download the rdma.json file:

   # wget http://git.infradead.org/users/hch/nvmetcli.git/blob_plain/0a6b088db2dc2e5de11e6f23f1e890e4b54fee64:/rdma.json

   Copy to Clipboard Toggle word wrap
3. Edit the rdma.json file and change the traddr value to 172.31.0.202.

4. Setup the controller by loading the NVMe controller configuration file:

   # nvmetcli restore rdma.json

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the nvme-cli tool:

   # dnf install nvme-cli

   Copy to Clipboard Toggle word wrap

2. Load the nvme-rdma module if it is not loaded:

   # modprobe nvme-rdma

   Copy to Clipboard Toggle word wrap

3. Discover available subsystems on the NVMe controller:

   # nvme discover -t rdma -a 172.31.0.202 -s 4420
   
   Discovery Log Number of Records 1, Generation counter 2
   =====Discovery Log Entry 0======
   trtype:  rdma
   adrfam:  ipv4
   subtype: nvme subsystem
   treq:    not specified, sq flow control disable supported
   portid:  1
   trsvcid: 4420
   subnqn:  testnqn
   traddr:  172.31.0.202
   rdma_prtype: not specified
   rdma_qptype: connected
   rdma_cms:    rdma-cm
   rdma_pkey: 0x0000

   Copy to Clipboard Toggle word wrap

4. Connect to the discovered subsystems:

   # nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420
   
   # lsblk
   NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   sda                            8:0    0 465.8G  0 disk
   ├─sda1                         8:1    0     1G  0 part /boot
   └─sda2                         8:2    0 464.8G  0 part
     ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
     ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
     └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
   nvme0n1
   
   # cat /sys/class/nvme/nvme0/transport
   rdma

   Copy to Clipboard Toggle word wrap

   Replace testnqn with the NVMe subsystem name.

   Replace 172.31.0.202 with the controller IP address.

   Replace 4420 with the port number.

Procedure

1. Install the nvme-cli utility:

   # dnf install nvme-cli

   Copy to Clipboard Toggle word wrap

   This creates the hostnqn file in the /etc/nvme/ directory. The hostnqn file identifies the NVMe host.

2. Find the World Wide Node Name (WWNN) and World Wide Port Name (WWPN) identifiers of the local and remote ports:

   # cat /sys/class/scsi_host/host*/nvme_info
   
   NVME Host Enabled
   XRI Dist lpfc0 Total 6144 IO 5894 ELS 250
   NVME LPORT lpfc0 WWPN x10000090fae0b5f5 WWNN x20000090fae0b5f5 DID x010f00 ONLINE
   NVME RPORT       WWPN x204700a098cbcac6 WWNN x204600a098cbcac6 DID x01050e TARGET DISCSRVC ONLINE
   
   NVME Statistics
   LS: Xmt 000000000e Cmpl 000000000e Abort 00000000
   LS XMIT: Err 00000000  CMPL: xb 00000000 Err 00000000
   Total FCP Cmpl 00000000000008ea Issue 00000000000008ec OutIO 0000000000000002
       abort 00000000 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000
   FCP CMPL: xb 00000000 Err 00000000

   Copy to Clipboard Toggle word wrap

   Using these host-traddr and traddr values, find the subsystem NVMe Qualified Name (NQN):

   # nvme discover --transport fc \
                   --traddr nn-0x204600a098cbcac6:pn-0x204700a098cbcac6 \
                   --host-traddr nn-0x20000090fae0b5f5:pn-0x10000090fae0b5f5
   
   Discovery Log Number of Records 2, Generation counter 49530
   =====Discovery Log Entry 0======
   trtype:  fc
   adrfam:  fibre-channel
   subtype: nvme subsystem
   treq:    not specified
   portid:  0
   trsvcid: none
   subnqn:  nqn.1992-08.com.netapp:sn.e18bfca87d5e11e98c0800a098cbcac6:subsystem.st14_nvme_ss_1_1
   traddr:  nn-0x204600a098cbcac6:pn-0x204700a098cbcac6

   Copy to Clipboard Toggle word wrap

   Replace nn-0x204600a098cbcac6:pn-0x204700a098cbcac6 with the traddr.

   Replace nn-0x20000090fae0b5f5:pn-0x10000090fae0b5f5 with the host-traddr.

3. Connect to the NVMe controller using the nvme-cli:

   # nvme connect --transport fc \
                  --traddr nn-0x204600a098cbcac6:pn-0x204700a098cbcac6 \
                  --host-traddr nn-0x20000090fae0b5f5:pn-0x10000090fae0b5f5 \
                  -n nqn.1992-08.com.netapp:sn.e18bfca87d5e11e98c0800a098cbcac6:subsystem.st14_nvme_ss_1_1 \
                  -k 5

   Copy to Clipboard Toggle word wrap
   Note

   If you see the keep-alive timer (5 seconds) expired! error when a connection time exceeds the default keep-alive timeout value, increase it using the -k option. For example, you can use, -k 7.

   Here,

   Replace nn-0x204600a098cbcac6:pn-0x204700a098cbcac6 with the traddr.

   Replace nn-0x20000090fae0b5f5:pn-0x10000090fae0b5f5 with the host-traddr.

   Replace nqn.1992-08.com.netapp:sn.e18bfca87d5e11e98c0800a098cbcac6:subsystem.st14_nvme_ss_1_1 with the subnqn.

   Replace 5 with the keep-alive timeout value in seconds.

Procedure

1. Install the nvme-cli utility:

   # dnf install nvme-cli

   Copy to Clipboard Toggle word wrap

   This creates the hostnqn file in the /etc/nvme/ directory. The hostnqn file identifies the NVMe host.

2. Reload the qla2xxx module:

   # modprobe -r qla2xxx
   # modprobe qla2xxx

   Copy to Clipboard Toggle word wrap

3. Find the World Wide Node Name (WWNN) and World Wide Port Name (WWPN) identifiers of the local and remote ports:

   # dmesg |grep traddr
   
   [    6.139862] qla2xxx [0000:04:00.0]-ffff:0: register_localport: host-traddr=nn-0x20000024ff19bb62:pn-0x21000024ff19bb62 on portID:10700
   [    6.241762] qla2xxx [0000:04:00.0]-2102:0: qla_nvme_register_remote: traddr=nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6 PortID:01050d

   Copy to Clipboard Toggle word wrap

   Using these host-traddr and traddr values, find the subsystem NVMe Qualified Name (NQN):

   # nvme discover --transport fc \
                   --traddr nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6 \
                   --host-traddr nn-0x20000024ff19bb62:pn-0x21000024ff19bb62
   
   Discovery Log Number of Records 2, Generation counter 49530
   =====Discovery Log Entry 0======
   trtype:  fc
   adrfam:  fibre-channel
   subtype: nvme subsystem
   treq:    not specified
   portid:  0
   trsvcid: none
   subnqn:  nqn.1992-08.com.netapp:sn.c9ecc9187b1111e98c0800a098cbcac6:subsystem.vs_nvme_multipath_1_subsystem_468
   traddr:  nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6

   Copy to Clipboard Toggle word wrap

   Replace nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6 with the traddr.

   Replace nn-0x20000024ff19bb62:pn-0x21000024ff19bb62 with the host-traddr.

4. Connect to the NVMe controller using the nvme-cli tool:

   # nvme connect --transport fc \
                   --traddr nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6 \
                   --host-traddr nn-0x20000024ff19bb62:pn-0x21000024ff19bb62 \
                   -n nqn.1992-08.com.netapp:sn.c9ecc9187b1111e98c0800a098cbcac6:subsystem.vs_nvme_multipath_1_subsystem_468\
                   -k 5

   Copy to Clipboard Toggle word wrap
   Note

   If you see the keep-alive timer (5 seconds) expired! error when a connection time exceeds the default keep-alive timeout value, increase it using the -k option. For example, you can use, -k 7.

   Here,

   Replace nn-0x203b00a098cbcac6:pn-0x203d00a098cbcac6 with the traddr.

   Replace nn-0x20000024ff19bb62:pn-0x21000024ff19bb62 with the host-traddr.

   Replace nqn.1992-08.com.netapp:sn.c9ecc9187b1111e98c0800a098cbcac6:subsystem.vs_nvme_multipath_1_subsystem_468 with the subnqn.

   Replace 5 with the keep-live timeout value in seconds.

Procedure

1. Install the nvme-cli tool:

   # dnf install nvme-cli

   Copy to Clipboard Toggle word wrap

   This tool creates the hostnqn file in the /etc/nvme/ directory, which identifies the NVMe host.

2. Find the nvme hostid and hostnqn:

   # cat /etc/nvme/hostnqn
   nqn.2014-08.org.nvmexpress:uuid:8ae2b12c-3d28-4458-83e3-658e571ed4b8
   
   # cat /etc/nvme/hostid
   09e2ce17-ccc9-412d-8dcf-2b0a1d581ee3

   Copy to Clipboard Toggle word wrap

   Use the hostid and hostnqn values to configure the NVMe/TCP controller.

3. Check the status of the controller:

   # nmcli device show ens6
   GENERAL.DEVICE:                         ens6
   GENERAL.TYPE:                           ethernet
   GENERAL.HWADDR:                         52:57:02:12:02:02
   GENERAL.MTU:                            1500
   GENERAL.STATE:                          30 (disconnected)
   GENERAL.CONNECTION:                     --
   GENERAL.CON-PATH:                       --
   WIRED-PROPERTIES.CARRIER:               on

   Copy to Clipboard Toggle word wrap

4. Configure the host network for a newly installed Ethernet controller with a static IP address:

   # nmcli connection add con-name ens6 ifname ens6 type ethernet ip4 192.168.101.154/24 gw4 192.168.101.1

   Copy to Clipboard Toggle word wrap

   Here, replace 192.168.101.154 with the host IP address.

   # nmcli connection mod ens6 ipv4.method manual
   # nmcli connection up ens6

   Copy to Clipboard Toggle word wrap

   Since a new network is created to connect the NVMe/TCP host to the NVMe/TCP controller, execute this step on the controller too.

Procedure

1. Load the nvme-tcp module if not already:

   # modprobe nvme-tcp

   Copy to Clipboard Toggle word wrap

2. Discover the available subsystems on the NVMe controller:

   # nvme discover --transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009
   
   Discovery Log Number of Records 2, Generation counter 7
   =====Discovery Log Entry 0======
   trtype:  tcp
   adrfam:  ipv4
   subtype: current discovery subsystem
   treq:	not specified, sq flow control disable supported
   portid:  2
   trsvcid: 8009
   subnqn:  nqn.2014-08.org.nvmexpress.discovery
   traddr:  192.168.101.55
   eflags:  not specified
   sectype: none
   =====Discovery Log Entry 1======
   trtype:  tcp
   adrfam:  ipv4
   subtype: nvme subsystem
   treq:	not specified, sq flow control disable supported
   portid:  2
   trsvcid: 8009
   subnqn:  nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
   traddr:  192.168.101.55
   eflags:  not specified
   sectype: none

   Copy to Clipboard Toggle word wrap

   Here, 192.168.101.55 is the NVMe/TCP controller IP address and 192.168.101.154 is the NVMe/TCP host IP address.

3. Configure the /etc/nvme/discovery.conf file to add the parameters used in the nvme discover command :

   # echo "--transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009" >> /etc/nvme/discovery.conf

   Copy to Clipboard Toggle word wrap

4. Connect the NVMe/TCP host to the controller system:

   # nvme connect-all

   Copy to Clipboard Toggle word wrap

5. Make the NVMe/TCP connection persistent:

   # systemctl enable nvmf-autoconnect.service

   Copy to Clipboard Toggle word wrap

Procedure

1. Generate an authentication secret:

   1. For the host:

      # hostkey=$(nvme gen-dhchap-key -n ${HOSTNQN})

      Copy to Clipboard Toggle word wrap

   2. For the subsystem:

      # ctrlkey= $(nvme gen-dhchap-key -n ${SUBSYSTEM})

      Copy to Clipboard Toggle word wrap

2. Configure the host for authentication:

   # nvme connect -t tcp -n ${SUBSYSTEM} -a ${TRADDR} -s 4420 --dhchap-secret=${hostkey} --dhchap-ctrl-secret=${ctrlkey}

   Copy to Clipboard Toggle word wrap

   This provides the authentication secrets to the nvme-connect utility so that it can authenticate and establish a connection to the target.

   • Optional: To enable automated logins, set up persistent NVMe fabrics configuration. To do so, add the --dhchap-secret and --dhchap-ctrl-secret parameters to /etc/nvme/discovery.conf or /etc/nvme/config.json.