13.2. Transferring Data Using RoCE

Procedure 13.1. Changing the Default RoCE Mode When the Hardware Is Already Running in Roce v2

1. Change into the /sys/kernel/config/rdma_cm directory to et the RoCE mode:

   ~]# cd /sys/kernel/config/rdma_cm

   Copy to Clipboard Toggle word wrap
2. Enter the ibstat command with an Ethernet network device to display the status. For example, for mlx5_0:

   ~]$ ibstat mlx5_0
       CA 'mlx5_0'
           CA type: MT4115
           Number of ports: 1
           Firmware version: 12.17.1010
           Hardware version: 0
           Node GUID: 0x248a0703004bf0a4
           System image GUID: 0x248a0703004bf0a4
           Port 1:
               State: Active
               Physical state: LinkUp
               Rate: 40
               Base lid: 0
               LMC: 0
               SM lid: 0
               Capability mask: 0x04010000
               Port GUID: 0x268a07fffe4bf0a4
               Link layer: Ethernet

   Copy to Clipboard Toggle word wrap
3. Create a directory for the mlx5_0 device:

   ~]# mkdir mlx5_0

   Copy to Clipboard Toggle word wrap
4. Display the RoCE mode in the default_roce_mode file in the tree format:

   ~]# cd mlx5_0

   Copy to Clipboard Toggle word wrap

   ~]$ tree
   └── ports
       └── 1
           ├── default_roce_mode
           └── default_roce_tos

   Copy to Clipboard Toggle word wrap

   ~]$ cat /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode
       RoCE v2

   Copy to Clipboard Toggle word wrap
5. Change the default RoCE mode:

   ~]# echo "RoCE v1" > /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode

   Copy to Clipboard Toggle word wrap
6. View the changes:

   ~]$ cat /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode
       RoCE v1

   Copy to Clipboard Toggle word wrap