Configure IPoIB

IP over InfiniBand (IPoIB) provides an IP network emulation layer on top of InfiniBand remote direct memory access (RDMA) networks. Unmodified applications can use IPoIB to transmit data over InfiniBand networks.

Note that the IPoIB performance is lower than if the application used RDMA natively.

Note

The Mellanox devices, starting from ConnectX-4 and above, on RHEL 8 and later use Enhanced IPoIB mode by default (datagram only). Connected mode is not supported on these devices.

The IPoIB communication modes
Copy link

An IPoIB device is configurable in either Datagram or Connected mode.

The difference is the type of queue pair the IPoIB layer attempts to open with the machine at the other end of the communication:

  • In the Datagram mode, the system opens an unreliable, disconnected queue pair.

    This mode does not support packages larger than Maximum Transmission Unit (MTU) of the InfiniBand link layer. During transmission of data, the IPoIB layer adds a 4-byte IPoIB header on top of the IP packet. As a result, the IPoIB MTU is 4 bytes less than the InfiniBand link-layer MTU. As 2048 is a common InfiniBand link-layer MTU, the common IPoIB device MTU in Datagram mode is 2044.

  • In the Connected mode, the system opens a reliable, connected queue pair.

    This mode allows messages larger than the InfiniBand link-layer MTU. The host adapter handles packet segmentation and reassembly. As a result, in the Connected mode, the messages sent from Infiniband adapters have no size limits. However, there are limited IP packets due to the data field and TCP/IP header field. For this reason, the IPoIB MTU in the Connected mode is 65520 bytes.

    The Connected mode has a higher performance but consumes more kernel memory.

Though a system is configured to use the Connected mode, a system still sends multicast traffic by using the Datagram mode because InfiniBand switches and fabric cannot pass multicast traffic in the Connected mode. Also, when the host is not configured to use the Connected mode, the system falls back to the Datagram mode.

While running an application that sends multicast data up to the MTU on the interface, configure the interface in Datagram mode or configure the application to cap the send size of a packet that will fit in datagram-sized packets.

IPoIB hardware address format and structure
Copy link

InfiniBand hardware addresses, often referred to as Globally Unique Identifiers (GUIDs), are unique identifiers assigned to each InfiniBand device, such as Host Channel Adapters (HCAs). These addresses are essential for routing, device identification, and network management.

IPoIB devices have a 20 byte hardware address that consists of the following parts:

  • The first 4 bytes are flags and queue pair numbers
  • The next 8 bytes are the subnet prefix

    The default subnet prefix is 0xfe:80:00:00:00:00:00:00. After the device connects to the subnet manager, the device changes this prefix to match with the configured subnet manager.

  • The last 8 bytes are the Globally Unique Identifier (GUID) of the InfiniBand port that attaches to the IPoIB device
Note

As the first 12 bytes can change, do not use them in the udev device manager rules.

Configure an IPoIB connection by using nmcli
Copy link

You can use the nmcli utility to create an IP over InfiniBand connection on the command line.

Before you begin
Copy link

  • An InfiniBand device is installed on the server
  • The corresponding kernel module is loaded

Procedure
Copy link

  1. Create the InfiniBand connection to use the mlx5_ib0 interface in the Connected transport mode and the maximum MTU of 65520 bytes: 
    # nmcli connection add type infiniband con-name mlx5_ib0 ifname mlx5_ib0 transport-mode Connected mtu 65520
  2. Set a P_Key, for example: 
    # nmcli connection modify mlx5_ib0 infiniband.p-key 0x8002
  3. Configure the IPv4 settings:
    • To use DHCP, enter: 
      # nmcli connection modify mlx5_ib0 ipv4.method auto

      Skip this step if ipv4.method is already set to auto (default).

    • To set a static IPv4 address, network mask, default gateway, DNS servers, and search domain, enter: 
      # nmcli connection modify mlx5_ib0 ipv4.method manual ipv4.addresses 192.0.2.1/24 ipv4.gateway 192.0.2.254 ipv4.dns 192.0.2.200 ipv4.dns-search example.com
  4. Configure the IPv6 settings:
    • To use stateless address autoconfiguration (SLAAC), enter: 
      # nmcli connection modify mlx5_ib0 ipv6.method auto

      Skip this step if ipv6.method is already set to auto (default).

    • To set a static IPv6 address, network mask, default gateway, DNS servers, and search domain, enter: 
      # nmcli connection modify mlx5_ib0 ipv6.method manual ipv6.addresses 2001:db8:1::1/64 ipv6.gateway 2001:db8:1::fffe ipv6.dns 2001:db8:1::ffbb ipv6.dns-search example.com
  5. To customize other settings in the profile, use the following command: 
    # nmcli connection modify mlx5_ib0 <setting><value>

    Enclose values with spaces or semicolons in quotes.

  6. Activate the profile: 
    # nmcli connection up mlx5_ib0

Results
Copy link

  • Use the ping utility to send ICMP packets to the remote host’s InfiniBand adapter, for example: 
    # ping -c5 192.0.2.2

Configure an IPoIB connection by using the network RHEL system role
Copy link

To configure IP over InfiniBand (IPoIB), create a NetworkManager connection profile. You can automate this process by using the network RHEL system role and remotely configure connection profiles on hosts defined in a playbook.

Before you begin
Copy link

  • You have prepared the control node and the managed nodes.
  • You are logged in to the control node as a user who can run playbooks on the managed nodes.
  • The account you use to connect to the managed nodes has sudo permissions for these nodes.
  • An InfiniBand device named mlx5_ib0 is installed in the managed nodes.
  • The managed nodes use NetworkManager to configure the network.

About this task
Copy link

You can use the network RHEL system role to configure IPoIB and, if a connection profile for the InfiniBand’s parent device does not exist, the role can create it as well.

Procedure
Copy link

  1. Create a playbook file, for example, ~/playbook.yml, with the following content: 
    ---
- name: Configure the network
  hosts: managed-node-01.example.com
  tasks:
    - name: IPoIB connection profile with static IP address settings
      ansible.builtin.include_role:
        name: redhat.rhel_system_roles.network
      vars:
        network_connections:
          # InfiniBand connection mlx5_ib0
          - name: mlx5_ib0
            interface_name: mlx5_ib0
            type: infiniband

          # IPoIB device mlx5_ib0.8002 on top of mlx5_ib0
          - name: mlx5_ib0.8002
            type: infiniband
            autoconnect: yes
            infiniband:
              p_key: 0x8002
              transport_mode: datagram
            parent: mlx5_ib0
            ip:
              address:
                - 192.0.2.1/24
                - 2001:db8:1::1/64
            state: up

    The settings specified in the example playbook include the following:

    type: <profile_type>
    Sets the type of the profile to create. The example playbook creates two connection profiles: One for the InfiniBand connection and one for the IPoIB device.
    parent: <parent_device>
    Sets the parent device of the IPoIB connection profile.
    p_key: <value>
    Sets the InfiniBand partition key. If you set this variable, do not set interface_name on the IPoIB device.
    transport_mode: <mode>
    Sets the IPoIB connection operation mode. You can set this variable to datagram (default) or connected.

    For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.network/README.md file on the control node.

  2. Validate the playbook syntax: 
    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook: 
    $ ansible-playbook ~/playbook.yml

Results
Copy link

  1. Display the IP settings of the mlx5_ib0.8002 device: 
    # ansible managed-node-01.example.com -m command -a 'ip address show mlx5_ib0.8002'
     
    managed-node-01.example.com | CHANGED | rc=0 >>
...
inet 192.0.2.1/24 brd 192.0.2.255 scope global noprefixroute ib0.8002
   valid_lft forever preferred_lft forever
inet6 2001:db8:1::1/64 scope link tentative noprefixroute
   valid_lft forever preferred_lft forever
  2. Display the partition key (P_Key) of the mlx5_ib0.8002 device: 
    # ansible managed-node-01.example.com -m command -a 'cat /sys/class/net/mlx5_ib0.8002/pkey'
     
    managed-node-01.example.com | CHANGED | rc=0 >>
0x8002
  3. Display the mode of the mlx5_ib0.8002 device: 
    # ansible managed-node-01.example.com -m command -a 'cat /sys/class/net/mlx5_ib0.8002/mode'
     
    managed-node-01.example.com | CHANGED | rc=0 >>
datagram

Configure an IPoIB connection by using nmstatectl
Copy link

You can use the declarative Nmstate API to configure an IP over InfiniBand (IPoIB). Nmstate ensures that the result matches the configuration file or rolls back the changes.

Before you begin
Copy link

  • An InfiniBand device is installed on the server.
  • The kernel module for the InfiniBand device is loaded.

Procedure
Copy link

  1. Create a YAML file, for example ~/create-IPoIB-profile.yml, with the following content: 
    interfaces:
- name: mlx5_ib0.8002
  type: infiniband
  state: up
  ipv4:
    enabled: true
    address:
    - ip: 192.0.2.1
      prefix-length: 24
    dhcp: false
  ipv6:
    enabled: true
    address:
    - ip: 2001:db8:1::1
      prefix-length: 64
    autoconf: false
    dhcp: false
  infiniband:
    base-iface: "mlx5_ib0"
    mode: datagram
    pkey: "0x8002"

routes:
  config:
  - destination: 0.0.0.0/0
    next-hop-address: 192.0.2.254
    next-hop-interface: mlx5_ib0.8002
  - destination: ::/0
    next-hop-address: 2001:db8:1::fffe
    next-hop-interface: mlx5_ib0.8002

    An IPoIB connection has now the following settings:

    • IPOIB device name: mlx5_ib0.8002
    • Base interface (parent): mlx5_ib0
    • InfiniBand partition key: 0x8002
    • Transport mode: datagram
    • Static IPv4 address: 192.0.2.1 with the /24 subnet mask
    • Static IPv6 address: 2001:db8:1::1 with the /64 subnet mask
    • IPv4 default gateway: 192.0.2.254
    • IPv6 default gateway: 2001:db8:1::fffe
  2. Apply the settings to the system: 
    # nmstatectl apply ~/create-IPoIB-profile.yml

Results
Copy link

  1. Display the IP settings of the mlx5_ib0.8002 device: 
    # ip address show mlx5_ib0.8002
     
    ...
inet 192.0.2.1/24 brd 192.0.2.255 scope global noprefixroute ib0.8002
   valid_lft forever preferred_lft forever
inet6 2001:db8:1::1/64 scope link tentative noprefixroute
   valid_lft forever preferred_lft forever
  2. Display the partition key (P_Key) of the mlx5_ib0.8002 device: 
    # cat /sys/class/net/mlx5_ib0.8002/pkey
     
    0x8002
  3. Display the mode of the mlx5_ib0.8002 device: 
    # cat /sys/class/net/mlx5_ib0.8002/mode
     
    datagram

Test an RDMA network by using iperf3 after IPoIB is configured
Copy link

The iperf3 utility measures RDMA and IP performance between two nodes.

Before you begin
Copy link

  • You have configured IPoIB on both hosts.

About this task
Copy link

In the following example, the large buffer size is used to perform a 60 seconds test to measure maximum throughput and fully use the bandwidth and latency between two hosts by using the iperf3 utility.

Procedure
Copy link

  1. To run iperf3 as a server on a system, define a time interval to provide periodic bandwidth updates -i to listen as a server -s that waits for the response of the client connection: 
    # iperf3 -i 5 -s
  2. To run iperf3 as a client on another system, define a time interval to provide periodic bandwidth updates -i to connect to the listening server -c of IP address 192.168.2.2 with -t time in seconds: 
    # iperf3 -i 5 -t 60 -c 192.168.2.2
  3. Use the following commands:
    1. Display test results on the system that acts as a server: 
      # iperf3 -i 10 -s
       
      -----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.3, port 22216
[5] local 192.168.2.2 port 5201 connected to 192.168.2.3 port 22218
[ID] Interval           Transfer     Bandwidth
[5]   0.00-10.00  sec  17.5 GBytes  15.0 Gbits/sec
[5]  10.00-20.00  sec  17.6 GBytes  15.2 Gbits/sec
[5]  20.00-30.00  sec  18.4 GBytes  15.8 Gbits/sec
[5]  30.00-40.00  sec  18.0 GBytes  15.5 Gbits/sec
[5]  40.00-50.00  sec  17.5 GBytes  15.1 Gbits/sec
[5]  50.00-60.00  sec  18.1 GBytes  15.5 Gbits/sec
[5]  60.00-60.04  sec  82.2 MBytes  17.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ID] Interval           Transfer     Bandwidth
[5]   0.00-60.04  sec  0.00 Bytes    0.00 bits/sec  sender
[5]   0.00-60.04  sec   107 GBytes  15.3 Gbits/sec  receiver
    2. Display test results on the system that acts as a client: 
      # iperf3 -i 1 -t 60 -c 192.168.2.2
       
      Connecting to host 192.168.2.2, port 5201
[4] local 192.168.2.3 port 22218 connected to 192.168.2.2 port 5201
[ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[4]   0.00-10.00  sec  17.6 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  10.00-20.00  sec  17.6 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  20.00-30.00  sec  18.4 GBytes  15.8 Gbits/sec    0   6.01 MBytes
[4]  30.00-40.00  sec  18.0 GBytes  15.5 Gbits/sec    0   6.01 MBytes
[4]  40.00-50.00  sec  17.5 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  50.00-60.00  sec  18.1 GBytes  15.5 Gbits/sec    0   6.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ID] Interval           Transfer     Bandwidth       Retr
[4]   0.00-60.00  sec   107 GBytes  15.4 Gbits/sec    0   sender
[4]   0.00-60.00  sec   107 GBytes  15.4 Gbits/sec        receiver