13.5.5. 连接到远程 Linux SRP 目标
SCSI RDMA 协议(SRP)是一个网络协议,它允许系统使用 RDMA 访问附加到另一个系统的 SCSI 设备。要允许 SRP 启动器在 SRP 目标端连接 SRP 目标,您必须为启动器中使用的主机通道适配器(HCA)端口添加访问控制列表(ACL)条目。
HCA 端口的 ACL ID 不唯一。ACL ID 取决于 HCA 的 GID 格式。使用同一驱动程序的 HCA(如
exampleib_qib )可以具有不同的 GID 格式。ACL ID 还取决于您启动连接请求的方式。
连接到远程 Linux SRP 目标:高级别概述
- 准备目标侧:
- 创建存储后端。例如,获取 /dev/sdc1 分区:
/> /backstores/block create vol1 /dev/sdc1 - 创建 SRP 目标:
/> /srpt create 0xfe80000000000000001175000077dd7e - 根据步骤中创建的后端创建一个 LUN:
/> /srpt/ib.fe80000000000000001175000077dd7e/luns create /backstores/block/vol1 - 为远程 SRP 客户端创建节点 ACL:
/> /srpt/ib.fe80000000000000001175000077dd7e/acls create 0x7edd770000751100001175000077d708请注意,srp_daemon的节点 ACL 不同。和ibsrpdm
- 为客户端发起与
srp_daemon 或的 SRP 连接:ibsrpdm[root@initiator]# srp_daemon -e -n -i qib0 -p 1 -R 60 &[root@initiator]# ibsrpdm -c -d /dev/infiniband/umad0 > /sys/class/infiniband_srp/srp-qib0-1/add_target - 可选。建议您使用不同的工具(如
lsscsi或dmesg)验证 SRP 连接。
过程 13.3. 使用 srp_daemon 或 ibsrpdm 连接到远程 Linux SRP 目标
- 对目标使用 ibstat 命令,以确定
状态和端口GUID值。HCA 必须处于Active状态。ACL ID 基于端口 GUID:[root@target]# ibstat CA 'qib0' CA type: InfiniPath_QLE7342 Number of ports: 1 Firmware version: Hardware version: 2 Node GUID: 0x001175000077dd7e System image GUID: 0x001175000077dd7e Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x0769086a Port GUID: 0x001175000077dd7e Link layer: InfiniBand - 获取 SRP 目标 ID,它基于 HCA 端口的 GUID。请注意,您需要将专用磁盘分区作为 SRP 目标的后端,如
/dev/sdc1。以下命令替换 fe80 的默认前缀,删除冒号,并在字符串的其余部分中添加新前缀:[root@target]# ibstatus | grep '<default-gid>' | sed -e 's/<default-gid>://' -e 's/://g' | grep 001175000077dd7e fe80000000000000001175000077dd7e - 使用
targetcli工具在块设备中创建 LUN vol1,创建一个 SRP 目标并导出 LUN:[root@target]# targetcli /> /backstores/block create vol1 /dev/sdc1 Created block storage object vol1 using /dev/sdc1. /> /srpt create 0xfe80000000000000001175000077dd7e Created target ib.fe80000000000000001175000077dd7e. /> /srpt/ib.fe80000000000000001175000077dd7e/luns create /backstores/block/vol1 Created LUN 0. /> ls / o- / ............................................................................. [...] o- backstores .................................................................. [...] | o- block ...................................................... [Storage Objects: 1] | | o- vol1 ............................... [/dev/sdc1 (77.8GiB) write-thru activated] | o- fileio ..................................................... [Storage Objects: 0] | o- pscsi ...................................................... [Storage Objects: 0] | o- ramdisk .................................................... [Storage Objects: 0] o- iscsi ................................................................ [Targets: 0] o- loopback ............................................................. [Targets: 0] o- srpt ................................................................. [Targets: 1] o- ib.fe80000000000000001175000077dd7e ............................... [no-gen-acls] o- acls ................................................................ [ACLs: 0] o- luns ................................................................ [LUNs: 1] o- lun0 ............................................... [block/vol1 (/dev/sdc1)] /> - 在启动器中使用 ibstat 命令检查状态是否为
Active,并确定端口 GUID:[root@initiator]# ibstat CA 'qib0' CA type: InfiniPath_QLE7342 Number of ports: 1 Firmware version: Hardware version: 2 Node GUID: 0x001175000077d708 System image GUID: 0x001175000077d708 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 2 LMC: 0 SM lid: 1 Capability mask: 0x07690868 Port GUID: 0x001175000077d708 Link layer: InfiniBand - 使用以下命令扫描,但不连接到远程 SRP 目标:目标 GUID 显示启动器已找到远程目标。ID 字符串显示远程目标是一个 Linux 软件目标(
ib_srpt.ko)。[root@initiator]# srp_daemon -a -o IO Unit Info: port LID: 0001 port GID: fe80000000000000001175000077dd7e change ID: 0001 max controllers: 0x10 controller[ 1] GUID: 001175000077dd7e vendor ID: 000011 device ID: 007322 IO class : 0100 ID: Linux SRP target service entries: 1 service[ 0]: 001175000077dd7e / SRP.T10:001175000077dd7e - 要验证 SRP 连接,请使用 lsscsi 命令列出 SCSI 设备,并在启动器连接到目标之前和之后比较 lsscsi 输出。
[root@initiator]# lsscsi [0:0:10:0] disk IBM-ESXS ST9146803SS B53C /dev/sda - 要在没有为 initiator 端口配置有效 ACL 的情况下连接到远程目标(应该会失败),请对
srp_daemonoribsrpdm使用以下命令:[root@initiator]# srp_daemon -e -n -i qib0 -p 1 -R 60 & [1] 4184[root@initiator]# ibsrpdm -c -d /dev/infiniband/umad0 > /sys/class/infiniband_srp/srp-qib0-1/add_target - dmesg 的输出显示 SRP 连接操作失败的原因。在后续步骤中,使用目标端的 dmesg 命令使情况变得明确。
[root@initiator]# dmesg -c [ 1230.059652] scsi host5: ib_srp: REJ received [ 1230.059659] scsi host5: ib_srp: SRP LOGIN from fe80:0000:0000:0000:0011:7500:0077:d708 to fe80:0000:0000:0000:0011:7500:0077:dd7e REJECTED, reason 0x00010006 [ 1230.073792] scsi host5: ib_srp: Connection 0/2 failed [ 1230.078848] scsi host5: ib_srp: Sending CM DREQ failed - 由于 LOGIN 失败,lsscsi 命令的输出与上一步中的输出相同。
[root@initiator]# lsscsi [0:0:10:0] disk IBM-ESXS ST9146803SS B53C /dev/sda - 在目标端使用 dmesg (
ib_srpt.ko)提供了有关 LOGIN 失败的原因。此外,输出中还包含srp_daemon提供的有效 ACL ID:0x7edd770000751100001175000077d708。[root@target]# dmesg [ 1200.303001] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x7edd770000751100:0x1175000077d708, t_port_id 0x1175000077dd7e:0x1175000077dd7e and it_iu_len 260 on port 1 (guid=0xfe80000000000000:0x1175000077dd7e) [ 1200.322207] ib_srpt Rejected login because no ACL has been configured yet for initiator 0x7edd770000751100001175000077d708. - 使用
targetcli工具添加有效的 ACL:[root@target]# targetcli targetcli shell version 2.1.fb41 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. /> /srpt/ib.fe80000000000000001175000077dd7e/acls create 0x7edd770000751100001175000077d708 Created Node ACL for ib.7edd770000751100001175000077d708 Created mapped LUN 0. - 验证 SRP LOGIN 操作:
- 等待 60 秒,以允许
srp_daemon重新登录:[root@initiator]# sleep 60 - 验证 SRP LOGIN 操作:
[root@initiator]# lsscsi [0:0:10:0] disk IBM-ESXS ST9146803SS B53C /dev/sda [7:0:0:0] disk LIO-ORG vol1 4.0 /dev/sdb - 对于 SRP 目标发现的内核日志,请使用:
[root@initiator]# dmesg -c [ 1354.182072] scsi host7: SRP.T10:001175000077DD7E [ 1354.187258] scsi 7:0:0:0: Direct-Access LIO-ORG vol1 4.0 PQ: 0 ANSI: 5 [ 1354.208688] scsi 7:0:0:0: alua: supports implicit and explicit TPGS [ 1354.215698] scsi 7:0:0:0: alua: port group 00 rel port 01 [ 1354.221409] scsi 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA [ 1354.229147] scsi 7:0:0:0: alua: Attached [ 1354.233402] sd 7:0:0:0: Attached scsi generic sg1 type 0 [ 1354.233694] sd 7:0:0:0: [sdb] 163258368 512-byte logical blocks: (83.5 GB/77.8 GiB) [ 1354.235127] sd 7:0:0:0: [sdb] Write Protect is off [ 1354.235128] sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 08 [ 1354.235550] sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 1354.255491] sd 7:0:0:0: [sdb] Attached SCSI disk [ 1354.265233] scsi host7: ib_srp: new target: id_ext 001175000077dd7e ioc_guid 001175000077dd7e pkey ffff service_id 001175000077dd7e sgid fe80:0000:0000:0000:0011:7500:0077:d708 dgid fe80:0000:0000:0000:0011:7500:0077:dd7e xyx