1. RoCE
1.1 Connectivity Requirements
RDMA requires the same class of network end to end, for example the same /16 subnet.
Single switch.
RoCE: host NIC ── Ethernet switch (PFC) ── peer NIC
Multiple switches.
RoCE: host NIC ── Leaf ── Spine ── Leaf ── peer NIC
When there are multiple hops, note:
| Explanation |
|---|
| Path | L2/L3 reachable is enough (same subnet or routable) |
| Lossless | PFC is recommended on every switch along the path; multi-hop relies more on ECN/DCQCN |
| MTU / rate | Consistent end to end |
| Anti-pattern | Do not improvise a path through “ordinary switches without PFC” |
It works across multiple switches if PFC is off, but under congestion performance drops noticeably or even becomes unstable — RDMA depends on a lossless network. RoCEv2 can be routed across L3; RoCEv1 is an L2 protocol, and you should not expect an ordinary router to forward it.
1
| apt install -y rdma-core ibverbs-utils perftest ethtool infiniband-diags
|
1.3 Devices
You need an Ethernet RDMA driver plus the corresponding netdev. Do not install opensm.
| Vendor | Typical modules | Notes |
|---|
| NVIDIA/Mellanox | mlx5_core mlx5_ib | ConnectX running RoCE; or use MLNX_OFED |
| Intel | irdma | E810 and other RoCE / iWARP |
| Broadcom | bnxt_en bnxt_re | RoCE |
| Cloud vendors | efa / erdma etc. | AWS, Alibaba Cloud |
1.4 Point-to-Point Testing
mlx5_0 port 1 ==> eth0x (Up)
mlx5_1 port 1 ==> eth1x (Up)
mlx5_bond_0 port 1 ==> bond0 (Up)
bond0 corresponds to mlx5_bond_0; the Linux slaves are eth2x + eth3x (not eth0x/eth1x).
1
| cat /sys/class/net/bond0/bonding/slaves
|
eth2x eth3x
RoCE runs RDMA over Ethernet and uses a GID (Global Identifier) to say “which address RDMA is sent from.” Every RDMA port has a GID table; each row has an index, so -x selects which row of that table to use, and -R uses RDMA_CM.
1
| ibv_devinfo -v -d mlx5_bond_0 | grep -A3 GID
|
Example for bond0:
1
2
3
4
5
6
7
8
9
10
| GID[ 0]: fe80:0000:0000:0000:0000:0000:0000:0001, RoCE v1
GID[ 1]: fe80::1, RoCE v2
GID[ 2]: fe80:0000:0000:0000:0000:0000:0000:0002, RoCE v1
GID[ 3]: fe80::2, RoCE v2
GID[ 4]: 0000:0000:0000:0000:0000:ffff:0a00:0001, RoCE v1
GID[ 5]: ::ffff:10.0.0.1, RoCE v2
GID[ 6]: 0000:0000:0000:0000:0000:ffff:0a00:0101, RoCE v1
GID[ 7]: ::ffff:10.0.1.1, RoCE v2
GID[ 8]: fe80:0000:0000:0000:0000:0000:0000:0003, RoCE v1
GID[ 9]: fe80::3, RoCE v2
|
The local bond0 address is 10.0.0.1, corresponding to GID index 5 (RoCE v2). -x selects the local GID, not the peer’s.
- Test connectivity from the client side
Prerequisites: -x selects the GID matching the local communication IP, and the peer IP goes at the end of the command.
Server
1
| ib_write_bw -d mlx5_bond_0 -x 5 --report_gbits
|
Client
1
| ib_write_bw -d mlx5_bond_0 -x 5 10.0.0.2 --report_gbits
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
| ---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_bond_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 128
CQ Moderation : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 5
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x1001 PSN 0xabc123 RKey 0x200000 VAddr 0x007f0000000000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:01
remote address: LID 0000 QPN 0x1002 PSN 0xdef456 RKey 0x200000 VAddr 0x007f0000001000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:02
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 23.12 23.09 0.044035
---------------------------------------------------------------------------------------
|
Server
1
| ib_write_lat -d mlx5_bond_0 -x 5 --report_gbits
|
Client
1
| ib_write_lat -d mlx5_bond_0 -x 5 10.0.0.2 --report_gbits
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| ---------------------------------------------------------------------------------------
RDMA_Write Latency Test
Dual-port : OFF Device : mlx5_bond_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF
ibv_wr* API : ON
TX depth : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 5
Max inline data : 220[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x1003 PSN 0xabc124 RKey 0x200001 VAddr 0x00560000000000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:01
remote address: LID 0000 QPN 0x1004 PSN 0xdef457 RKey 0x200002 VAddr 0x00560000001000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:02
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 3.40 6.22 3.48 3.49 0.03 3.71 6.22
---------------------------------------------------------------------------------------
|
2. Soft RoCE
2.1 Connectivity Requirements
RDMA requires the same class of network end to end. Soft RoCE runs on an ordinary Ethernet port, so the topology is the same as RoCE, but in development and integration scenarios the switch does not have to enforce PFC.
Single switch.
host NIC ── Ethernet switch ── peer NIC
Multiple switches.
host NIC ── Leaf ── Spine ── Leaf ── peer NIC
When there are multiple hops, note:
| Explanation |
|---|
| Path | L2/L3 reachable is enough (same subnet or routable) |
| NIC | The netdev bound to RXE must have an IP configured first |
| MTU / rate | Consistent end to end |
| Performance | No hardware offload; unsuited to production high-performance network needs |
1
| apt install -y rdma-core ibverbs-utils perftest ethtool infiniband-diags
|
2.3 Devices
Without an RDMA NIC, use the kernel software stack to emulate RoCE on an ordinary Ethernet port (RXE, module rdma_rxe). Suitable for development and integration; performance is far below hardware RoCE and it is generally not used in production.
| Item | Explanation |
|---|
| Kernel module | rdma_rxe |
| Device names | rxe0, rxe1, … |
| Bound to | An ordinary netdev that already has an IP (e.g. eth0) |
| vs. hardware RoCE | Similar protocol, no hardware offload, high CPU cost |
2.4 Point-to-Point Testing
Both ends must first create the RXE, bound to an ordinary NIC with an IP (in the example below, eth0 → rxe0).
1
| rdma link add rxe0 type rxe netdev eth0
|
1
2
3
| device node GUID
------ ----------------
rxe0 0000000000000001
|
1
| ibv_devinfo -v -d rxe0 | grep -A3 GID
|
GID[ 0]: fe80:0000:0000:0000:0000:0000:0000:0001
GID[ 1]: 0000:0000:0000:0000:0000:ffff:0a00:0001
GID[ 2]: 0000:0000:0000:0000:0000:ffff:0a00:0101
The local address 10.0.0.1 corresponds to GID[1] (0a00:0001), so the client uses -x 1; the peer IP 10.0.0.2 goes at the end of the command.
- Test connectivity from the client side
Server
1
| ib_write_bw -d rxe0 -x 1 --report_gbits
|
Client
1
| ib_write_bw -d rxe0 -x 1 10.0.0.2 --report_gbits
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| ---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : rxe0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0011 PSN 0x100001 RKey 0x000200 VAddr 0x007f0000000000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:01
remote address: LID 0000 QPN 0x0011 PSN 0x100002 RKey 0x000200 VAddr 0x007f0000001000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:02
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 0.16 0.14 0.000275
---------------------------------------------------------------------------------------
|
Measured with perf, the bandwidth between these two nodes is 3.38 Gbits/sec, far higher than Soft RoCE.
Server
1
| ib_write_lat -d rxe0 -x 1 --report_gbits
|
Client
1
| ib_write_lat -d rxe0 -x 1 10.0.0.2 --report_gbits
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| ---------------------------------------------------------------------------------------
RDMA_Write Latency Test
Dual-port : OFF Device : rxe0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0011 PSN 0x100003 RKey 0x000210 VAddr 0x00550000000000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:01
remote address: LID 0000 QPN 0x0011 PSN 0x100004 RKey 0x000210 VAddr 0x00550000001000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:00:02
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 383.61 1007.44 508.28 535.29 95.27 850.50 1007.44
---------------------------------------------------------------------------------------
|
3. InfiniBand
3.1 Connectivity Requirements
InfiniBand must traverse an IB fabric end to end. The whole network is managed by a Subnet Manager; nodes within the same IB subnet only become Active once their ports are assigned a LID.
Single switch.
IB: host HCA ── IB switch (SM) ── peer HCA
Multiple switches.
IB: host HCA ── Leaf ── Spine ── Leaf ── peer HCA
When there are multiple hops, note:
| Explanation |
|---|
| Path | The SM manages the whole network and forwards by LID |
| Lossless | The IB link layer does credit-based flow control itself |
| MTU / rate | Consistent end to end |
| Anti-pattern | Do not mix in Ethernet switches |
1
| apt install -y rdma-core ibverbs-utils perftest ethtool infiniband-diags
|
3.3 Devices
| Vendor | Typical modules | Notes |
|---|
| NVIDIA/Mellanox | mlx5_core mlx5_ib | ConnectX running native IB; MLNX_OFED commonly used |
| Other IB HCAs | See vendor docs | Module names vary by card |
A small network needs a Subnet Manager (opensm or a switch SM); IPoIB is optional: modprobe ib_ipoib.
1
| lsmod | grep -iE 'mlx5|ib_ipoib|ib_core'
|
device node GUID
------ ----------------
mlx5_0 0000000000000001
mlx5_1 0000000000000002
mlx5_4 0000000000000003
mlx5_5 0000000000000004
mlx5_bond_0 0000000000000005
3.4 Inspecting the Link Topology
On a machine within the same IB subnet, running sminfo returns something like:
1
| sminfo: sm lid 1 sm guid 0x0000000000000001, activity count 237173850 priority 0 state 3 SMINFO_MASTER
|
If both ends see the same sm lid and sm guid, and state is SMINFO_MASTER, they are managed by the same Subnet Manager and are in the same IB subnet — the prerequisite for interoperability, along with conditions such as the local port being Active and matching PKey/partition. activity count changes over time and need not match.
vendid=0x2c9
devid=0xd2f0
sysimgguid=0x0000000000001001
switchguid=0x0000000000001001(0000000000001001)
Switch 41 "S-0000000000001001" # "Quantum Mellanox Technologies" base port 0 lid 141 lmc 0
[19] "H-0000000000002001"[1](0000000000002001) # "node-01 mlx5_4" lid 185 4xHDR
[20] "H-0000000000002002"[1](0000000000002002) # "node-01 mlx5_5" lid 187 4xHDR
[21] "S-0000000000001002"[37] # "MF0;sw-agg-1:MQM8700/U1" lid 50 4xHDR
[23] "S-0000000000001002"[39] # "MF0;sw-agg-1:MQM8700/U1" lid 50 4xHDR
The command prints the entire fabric visible to the SM. The above is a partial view of one Quantum switch (lid 141): ports [19]/[20] connect to two HCAs on the same host node-01 (mlx5_4 / mlx5_5), both at 4xHDR; [21]/[23] connect to the upstream aggregation switch sw-agg-1. Good for checking “what is this port connected to” from the switch’s point of view.
CA: Mellanox Technologies Aggregation Node:
0x0000000000003001 13 1[ ] ==( 4X 53.125 Gbps Active/ LinkUp)==> 141 41[ ] "Quantum Mellanox Technologies" ( )
CA: node-01 mlx5_5:
0x0000000000002002 187 1[ ] ==( 4X 53.125 Gbps Active/ LinkUp)==> 141 20[ ] "Quantum Mellanox Technologies" ( )
Switch: 0x0000000000001003 Quantum Mellanox Technologies:
49 1[ ] ==( 4X 53.125 Gbps Active/ LinkUp)==> 173 1[ ] "node-02 mlx5_3" ( )
49 2[ ] ==( 4X 53.125 Gbps Active/ LinkUp)==> 38 1[ ] "node-03 mlx5_5" ( )
Better suited to day-to-day troubleshooting than ibnetdiscover. Each line reads “local lid/port ==( width rate state )==> peer lid/port”. Here Active/LinkUp means both the logical and the physical link are healthy; 4X 53.125 Gbps corresponds to the HDR rate. In the example, node-01 mlx5_5 (lid 187) connects to port 20 of the Quantum (lid 141), matching [20] in the topology above.
Switch : 0x0000000000001001 ports 41 "Quantum Mellanox Technologies" base port 0 lid 141 lmc 0
Switch : 0x0000000000001004 ports 41 "MF0;sw-agg-5:MQM8700/U1" enhanced port 0 lid 55 lmc 0
Quickly count how many switches are in the fabric and their lids. Here there is at least the local-side Quantum (lid 141) and the upstream sw-agg-5 (lid 55); ports 41 includes the switch management port (port 0), while the service ports are usually 40.
Ca : 0x0000000000003001 ports 1 "Mellanox Technologies Aggregation Node"
Ca : 0x0000000000002002 ports 1 "node-01 mlx5_5"
Lists the CAs (Channel Adapters) registered with the SM. You can see both ordinary compute-node HCAs (node-01 mlx5_5) and the switch-side Aggregation Node; cross-checking against iblinkinfo confirms whether all hosts are attached to the fabric.
- More complete diagnostics (MLNX_OFED / DOCA)
1
| ibdiagnet -o ./ib-report
|
-I- Stages Status Report:
STAGE Errors Warnings
Bad GUIDs/LIDs Check 0 0
Link State Active Check 0 0
General Devices Info Report 0 0
Performance Counters Report 0 0
Partitions Check 0 0
IPoIB Subnets Check 0 1
Check whether any node’s state is abnormal; the detailed report is under ./ib-report.
3.5 Point-to-Point Testing
This uses TCP handshake mode: QP information is exchanged over TCP while RDMA data still travels over the IB link.
State: Active
Base lid: 1
Port State=Active.
SM reachable.
1
2
| ip -4 addr | grep 10.0.0
ping -c 3 10.0.0.2
|
Server
1
| ib_write_bw -d mlx5_0 --report_gbits
|
Client
1
| ib_write_bw -d mlx5_0 --report_gbits 10.0.0.2
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| ---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 128
CQ Moderation : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1 QPN 0x1001 PSN 0xabc123 RKey 0x210000 VAddr 0x007f0000000000
remote address: LID 0x2 QPN 0x1002 PSN 0xdef456 RKey 0x216000 VAddr 0x007f0000001000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 179.98 179.93 0.343193
---------------------------------------------------------------------------------------
|
Server
1
| ib_write_lat -d mlx5_0 --report_gbits
|
Client
1
| ib_write_lat -d mlx5_0 --report_gbits 10.0.0.2
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| ---------------------------------------------------------------------------------------
RDMA_Write Latency Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF
ibv_wr* API : ON
TX depth : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 220[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x1 QPN 0x1003 PSN 0xabc124 RKey 0x1ec000 VAddr 0x00560000000000
remote address: LID 0x2 QPN 0x1004 PSN 0xdef457 RKey 0x1e5000 VAddr 0x00560000001000
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 2.44 8.29 2.56 2.56 0.10 2.58 8.29
---------------------------------------------------------------------------------------
|
