Este contenido no está disponible en el idioma seleccionado.
13.3. Configuring Soft-RoCE
RoCE can be implemented both in the hardware and in the software. Soft-RoCE is the software implementation of the RDMA transport.
Prerequisites
Since Red Hat Enterprise Linux 7.4, the Soft-RoCE driver is already merged into the kernel. The user-space driver also is merged into the rdma-core package. Soft-RoCE is also known as RXE. To start, stop and configure RXE, use the
rxe_cfg
script. To view options for rxe_cfg
, enter rxe_cfg help
.
Procedure 13.2. Configuring Soft-RoCE
- As the
root
user, enter the following command to display the current configuration status of RXE:~]#
rxe_cfg
rdma_rxe module not loaded Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1yes
igb mlx4_1 no mlx4_en mlx4_2 no mlx4_en - To load the RXE kernel module and start RXE, enter as
root
:~]#
Optionally, to verify that the RXE kernel module is loaded, enter:rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1yes
igb mlx4_1 no mlx4_en mlx4_2 no mlx4_en~]#
lsmod |grep rdma_rxe
rdma_rxe 111129 0 ip6_udp_tunnel 12755 1 rdma_rxe udp_tunnel 14423 1 rdma_rxe ib_core 236827 15 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_rxe,rdma_ucm,ib_ipoib,ib_isert - Before adding a new RXE device over an Ethernet interface, the corresponding interface should be opened and has a valid IP address assigned. To add a new RXE device, for example igb_1:
~]#
rxe_cfg add igb_1
~]#
The rxe0 in the RDEV column indicates that rxe is enabled for the igb_1 device.rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1 yes igb rxe0 1024 (3) mlx4_1 no mlx4_en mlx4_2 no mlx4_en - To verify the status of an RXE device, use the
ibv_devices
command:~]#
Alternatively, enter theibv_devices
device node GUID ------ ---------------- mlx4_0 0002c90300b3cff0 rxe0 a2369ffffe018294ibstat
for a detailed status:~]#
ibstat
rxe0 CA 'rxe0' CA type: Number of ports: 1 Firmware version: Hardware version: Node GUID: 0xa2369ffffe018294 System image GUID: 0x0000000000000000 Port 1: State: Active Physical state: LinkUp Rate: 2.5 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00890000 Port GUID: 0xa2369ffffe018294 Link layer: Ethernet
Removing an RXE device
If you want to remove an RXE device, enter:
~]# rxe_cfg remove igb_1
Verifying Connectivity of an RXE device
The following examples show how to verify connectivity of an RXE device on the server and client side.
Example 13.1. Verifying Connectivity of an RXE device on the Server Side
~]$ ibv_rc_pingpong -d rxe0 -g 0
local address: LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
remote address: LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
8192000 bytes in 0.05 seconds = 1244.06 Mbit/sec
1000 iters in 0.05 seconds = 52.68 usec/iter
Example 13.2. Verifying Connectivity of an RXE device on the Client Side
~]$ ibv_rc_pingpong -d rxe0 -g 0 172.31.40.4
local address: LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
remote address: LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
8192000 bytes in 0.05 seconds = 1245.72 Mbit/sec
1000 iters in 0.05 seconds = 52.61 usec/iter