NFS Over RDMA

1. Prerequisites

The storage and the nodes that use it form an RDMA network
The RDMA devices are configured with IP addresses
If you are using Mellanox NICs, add the -with-nfsrdma parameter when installing the driver

1
./mlnxofedinstall --with-nfsrdma

2. Starting the Server

Install dependencies

1
apt install nfs-kernel-server rdma-core -y

Load the kernel module

1
modprobe svcrdma

Add the port to portlist

1
echo 'rdma 20049' | tee /proc/fs/nfsd/portlist

Export directory

1
mkdir /data1/nfs

1
2
3
vim /etc/exports

/data1/nfs  *(rw,sync,no_root_squash,no_all_squash)

Start the NFS service

1
systemctl start nfs-server.service

Check the export points

1
2
3
exportfs -v

/data1/nfs      <world>(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash

3. Mounting on the Client

Install dependencies

1
apt install nfs-common rdma-core -y

Load the kernel module

1
modprobe rpcrdma

Create the mount directory

1
mkdir /data1/nfs

Plain NFS mount

1
mount -t nfs x.x.x.x:/data1/nfs /data1/nfs

NFS Over RDMA mount

1
mount -o proto=rdma,port=20049,vers=4 10.10.x.x:/data1/nfs /data1/nfs

Note that a single host cannot mount both plain NFS and NFS Over RDMA at the same time.

4. Performance Testing

4.1 Network Performance

Ordinary NIC

1
iperf3 -s

1
iperf3 -c x.x.x.x

1
2
3
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.2 GBytes  23.4 Gbits/sec  4104             sender
[  5]   0.00-10.05  sec  27.2 GBytes  23.3 Gbits/sec                  receiver

RDMA NIC

1
ib_write_bw -d mlx5_0 -x3

1
ib_write_bw -d mlx5_0 -x3 x.x.x.x --report_gbits

1
2
3
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 3100.000000 != 2101.000000. CPU Frequency is not max.
 65536      5000             389.06             389.00             0.741968

4.2 4K Random Read/Write

Local disk 4K random read/write

1
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Write_Testing -filename=/data1/fiofile

1
  WRITE: bw=1128MiB/s (1182MB/s), 1128MiB/s-1128MiB/s (1182MB/s-1182MB/s), io=10.0GiB (10.7GB), run=9081-9081msec

1
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Read_Testing -filename=/data1/fiofile

1
   READ: bw=1441MiB/s (1511MB/s), 1441MiB/s-1441MiB/s (1511MB/s-1511MB/s), io=10.0GiB (10.7GB), run=7107-7107msec

RDMA 4K random read/write

1
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Write_Testing -filename=/data1/nfs/fiofile

1
  WRITE: bw=525MiB/s (551MB/s), 525MiB/s-525MiB/s (551MB/s-551MB/s), io=10.0GiB (10.7GB), run=19488-19488msec

1
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Read_Testing -filename=/data1/nfs/fiofile

1
   READ: bw=674MiB/s (707MB/s), 674MiB/s-674MiB/s (707MB/s-707MB/s), io=10.0GiB (10.7GB), run=15191-15191msec

NFS 4K random read/write

1
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Write_Testing -filename=/data1/nfs/fiofile

1
  WRITE: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=10.0GiB (10.7GB), run=35196-35196msec

1
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=10g -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Read_Testing -filename=/data1/nfs/fiofile

1
   READ: bw=311MiB/s (326MB/s), 311MiB/s-311MiB/s (326MB/s-326MB/s), io=10.0GiB (10.7GB), run=32909-32909msec

4.3 128K Read/Write

Local disk 128K read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/fiofiledir

1
  WRITE: bw=3842MiB/s (4029MB/s), 3842MiB/s-3842MiB/s (4029MB/s-4029MB/s), io=113GiB (121GB), run=30009-30009msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/fiofiledir

1
   READ: bw=6503MiB/s (6818MB/s), 6503MiB/s-6503MiB/s (6818MB/s-6818MB/s), io=191GiB (205GB), run=30006-30006msec

RDMA 128K read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
  WRITE: bw=2814MiB/s (2951MB/s), 2814MiB/s-2814MiB/s (2951MB/s-2951MB/s), io=82.5GiB (88.6GB), run=30012-30012msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
   READ: bw=16.5GiB/s (17.7GB/s), 16.5GiB/s-16.5GiB/s (17.7GB/s-17.7GB/s), io=496GiB (533GB), run=30003-30003msec

NFS 128K read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
  WRITE: bw=2249MiB/s (2358MB/s), 2249MiB/s-2249MiB/s (2358MB/s-2358MB/s), io=65.9GiB (70.8GB), run=30018-30018msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
   READ: bw=2556MiB/s (2680MB/s), 2556MiB/s-2556MiB/s (2680MB/s-2680MB/s), io=74.9GiB (80.4GB), run=30012-30012msec

4.4 4M Read/Write

Local disk 4M read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/fiofiledir

1
  WRITE: bw=3838MiB/s (4025MB/s), 3838MiB/s-3838MiB/s (4025MB/s-4025MB/s), io=113GiB (122GB), run=30242-30242msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/fiofiledir

1
   READ: bw=5510MiB/s (5778MB/s), 5510MiB/s-5510MiB/s (5778MB/s-5778MB/s), io=162GiB (174GB), run=30167-30167msec

RDMA 4M read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
  WRITE: bw=3840MiB/s (4027MB/s), 3840MiB/s-3840MiB/s (4027MB/s-4027MB/s), io=113GiB (121GB), run=30159-30159msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
   READ: bw=42.4GiB/s (45.6GB/s), 42.4GiB/s-42.4GiB/s (45.6GB/s-45.6GB/s), io=1274GiB (1368GB), run=30018-30018msec

NFS 4M read/write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
  WRITE: bw=2520MiB/s (2643MB/s), 2520MiB/s-2520MiB/s (2643MB/s-2643MB/s), io=74.2GiB (79.7GB), run=30165-30165msec

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs

1
   READ: bw=2754MiB/s (2888MB/s), 2754MiB/s-2754MiB/s (2888MB/s-2888MB/s), io=81.5GiB (87.5GB), run=30312-30312msec

4.5 Summary

4K random read/write performance

Test Type	Direction	Local Disk (MiB/s)	RDMA (MiB/s)	NFS (MiB/s)
Random write	Write	1128	525	291
Random read	Read	1441	674	311

128K read/write performance

Test Type	Direction	Local Disk (MiB/s)	RDMA (MiB/s)	NFS (MiB/s)
Sequential write	Write	3842	2814	2249
Sequential read	Read	6503	16.5 GiB/s	2556

4M read/write performance

Test Type	Direction	Local Disk (MiB/s)	RDMA (MiB/s)	NFS (MiB/s)
Sequential write	Write	3838	3840	2520
Sequential read	Read	5510	42.4 GiB/s	2754

5. Conclusion

This post mainly introduces the configuration and performance testing of NFS Over RDMA. Judging from the test results:

Whether for large or small files, NFS Over RDMA performs better than NFS
The larger the file, the closer NFS Over RDMA’s write performance gets to the local disk

NFS Over RDMA’s performance reading large files far exceeds the tests run on the local host; changing either the duration or the concurrency yields consistent results, which is a bit hard to explain. But the monitoring shows that the RDMA data transfer bandwidth matches the FIO test results, which seems to suggest the results are correct.

Could this be caused by NFS Over RDMA’s caching mechanism?

6. References

https://www.cs.cmu.edu/~wtantisi/files/nfs-pdl08-poster.pdf