1. RoCE 1.1 Connectivity Requirements RDMA requires the same class of network end to end, for example the same /16 subnet. Single switch. RoCE: host NIC ── Ethernet switch (PFC) ── peer NIC Multiple switches. RoCE: host NIC ── Leaf ── Spine ── Leaf ── peer NIC When there are multiple hops, note:

In a PD disaggregated deployment, heterogeneous GPU models are often used to deploy the model across machines, which multiplies the cross-machine communication pressure. RDMA devices are usually brought in to accelerate kvcache transfer between nodes so as to achieve a lower FTTL. This post describes how to test eRDMA devices

1. Background While testing the performance of NFS Over RDMA, I found that reading a 4M file could actually reach 45GB/s. 1 2 3 fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs READ: bw=42.3GiB/s (45.4GB/s), 42.3GiB/s-42.3GiB/s (45.4GB/s-45.4GB/s), io=1269GiB (1363GB), run=30019-30019msec Yet the disk’s 4M multi-threaded read performance is only 6 GB/s.

1. Prerequisites The storage and the nodes that use it form an RDMA network The RDMA devices are configured with IP addresses If you are using Mellanox NICs, add the -with-nfsrdma parameter when installing the driver 1 ./mlnxofedinstall --with-nfsrdma 2. Starting the Server Install dependencies 1 apt install nfs-kernel-server rdma-core -y Load the kernel module 1 modprobe svcrdma Add the port to portlist 1 echo 'rdma 20049' | tee /proc/fs/nfsd/portlist Export directory 1 mkdir /data1/nfs 1 2 3 vim /etc/exports /data1/nfs *(rw,sync,no_root_squash,no_all_squash) Start the NFS service 1 systemctl start nfs-server.

1. What RDMA Is RDMA (Remote Direct Memory Access) is a technology created to solve the delay of server-side data processing in network transmission. With TCP/IP transmission, data passes through the network stack, then through the NIC to be sent; after the receiver receives it, the data is reassembled according to sequence numbers.