3FS: Some Performance Tests

1. Hardware Tests

There are two NVMe SSDs (SAMSUNG MZQL27T6HBLA-00A07) and a 3FS cluster made up of two nodes, each with 4 IB ports.

1.1 Single Thread

Disk FIO 128 K, read

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 1 (f=1): [R(1)][100.0%][r=2211MiB/s][r=17.7k IOPS][eta 00m:00s]

Disk FIO 128 K, write

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 1 (f=1): [W(1)][100.0%][w=3703MiB/s][w=29.6k IOPS][eta 00m:00s]

Disk FIO 4 M, read

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 1 (f=1): [R(1)][100.0%][r=6294MiB/s][r=1573 IOPS][eta 00m:00s]

Disk FIO 4 M, write

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 1 (f=1): [W(1)][100.0%][w=3872MiB/s][w=967 IOPS][eta 00m:00s]

Summary

Device single thread

Block Size	Operation	Read/Write Rate	IOPS
128 K	Read	2211 MiB/s	17.7k
128 K	Write	3703 MiB/s	29.6k
4 M	Read	6294 MiB/s	1573
4 M	Write	3872 MiB/s	967

1.2 Multithread

Disk FIO 128 K, read

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 128 (f=128): [R(128)][100.0%][r=6237MiB/s][r=49.9k IOPS][eta 00m:00s]

Disk FIO 128 K, write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 128 (f=128): [W(128)][100.0%][w=3873MiB/s][w=31.0k IOPS][eta 00m:00s]

Disk FIO 4 M, read

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 128 (f=128): [R(128)][100.0%][r=5832MiB/s][r=1458 IOPS][eta 00m:00s]

Disk FIO 4 M, write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio

1
Jobs: 128 (f=128): [W(128)][100.0%][w=3768MiB/s][w=941 IOPS][eta 00m:00s]

Summary

Device multithread

Block Size	Operation	Read/Write Rate	IOPS
128 K	Read	6237 MiB/s	49.9k
128 K	Write	3873 MiB/s	31.0k
4 M	Read	5832 MiB/s	1458
4 M	Write	3768 MiB/s	941

1.3 IB Devices

Device list, 4 ports

1
2
3
4
5
6
ibdev2netdev

mlx5_0 port 1 ==> ibp25s0 (Up)
mlx5_1 port 1 ==> ibp59s0 (Up)
mlx5_2 port 1 ==> ibp155s0 (Up)
mlx5_3 port 1 ==> ibp187s0 (Up)

IB link rate 53.125 Gbps

1
2
3
4
iblinkinfo  |grep h100

          65   19[  ] ==( 4X        53.125 Gbps Active/  LinkUp)==>     172    1[  ] "h100-03 mlx5_0" ( )
          ...

IB bandwidth test

Run on one of the machines

1
ib_write_bw -d mlx5_0 -x3

Run on the other machine

1
ib_write_bw -d mlx5_0 -x3 x.x.x.x --report_gbits

1
2
3
4
5
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2101.000000 != 3100.000000. CPU Frequency is not max.
 65536      5000             195.48             195.46             0.372806
---------------------------------------------------------------------------------------

A single network port can provide 195 Gbps of bandwidth.

2 3FS Mounted Host Tests

2.1 Single Thread

Disk FIO 128 K, read

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 1 (f=1): [R(1)][100.0%][r=937MiB/s][r=7497 IOPS][eta 00m:00s]

Disk FIO 128 K, write

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 1 (f=1): [W(1)][100.0%][w=25.6MiB/s][w=205 IOPS][eta 00m:00s]

Disk FIO 4 M, read

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 1 (f=1): [R(1)][100.0%][r=7736MiB/s][r=1934 IOPS][eta 00m:00s]

Disk FIO 4 M, write

1
fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 1 (f=1): [W(1)][100.0%][w=1233MiB/s][w=308 IOPS][eta 00m:00s]

Summary

3FS single thread

Block Size	Operation	Read/Write Rate	IOPS
128 K	Read	937 MiB/s	7497
128 K	Write	25.6 MiB/s	205
4 M	Read	7736 MiB/s	1934
4 M	Write	1233 MiB/s	308

2.2 Multithread

Disk FIO 128 K, read

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 128 (f=128): [R(128)][100.0%][r=6941MiB/s][r=55.5k IOPS][eta 00m:00s]

Disk FIO 128 K, write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 128 (f=128): [W(128)][100.0%][w=209MiB/s][w=1675 IOPS][eta 00m:00s]

Disk FIO 4 M, read

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 128 (f=128): [R(128)][100.0%][r=11.9GiB/s][r=3056 IOPS][eta 00m:00s]

Disk FIO 4 M, write

1
fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=write -bs=4M --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/3fs/stage/fio

1
Jobs: 128 (f=128): [W(128)][100.0%][w=1648MiB/s][w=412 IOPS][eta 00m:00s]

Summary

3FS multithread

Block Size	Operation	Read/Write Rate	IOPS
128 K	Read	6941 MiB/s	55.5k
128 K	Write	209 MiB/s	1675
4 M	Read	11.9 GiB/s	3056
4 M	Write	1648 MiB/s	412

2.3 3FS vs. Device

Single thread

Block Size	Operation	Device Read/Write Rate	Device IOPS	3FS Read/Write Rate	3FS IOPS
128 K	Read	2211 MiB/s	17.7k	937 MiB/s	7497
128 K	Write	3703 MiB/s	29.6k	25.6 MiB/s	205
4 M	Read	6294 MiB/s	1573	7736 MiB/s	1934
4 M	Write	3872 MiB/s	967	1233 MiB/s	308

Multithread

Block Size	Operation	Device Read/Write Rate	Device IOPS	3FS Read/Write Rate	3FS IOPS
128 K	Read	6237 MiB/s	49.9k	6941 MiB/s	55.5k
128 K	Write	3873 MiB/s	31.0k	209 MiB/s	1675
4 M	Read	5832 MiB/s	1458	11.9 GiB/s	3056
4 M	Write	3768 MiB/s	941	1648 MiB/s	412

3FS is suited to multithreaded, large-block read and write operations; its performance on single-threaded, small-block reads and writes is poor.

3. Checkpoint Storage Speed Test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import os
import time
import torch
import argparse

def generate_dummy_checkpoint(size_mb):
    """
    生成一个指定大小的模拟 checkpoint 文件。
    size_mb: 文件大小，以 MB 为单位。
    """
    # 1 MB = 1024 * 1024 bytes
    size_bytes = size_mb * 1024 * 1024
    dummy_data = torch.rand(size_bytes // 4)  # 随机生成数据，假设每个浮点数占 4 字节
    return {"dummy_data": dummy_data}

def save_checkpoint(checkpoint, save_dir):
    """
    将 checkpoint 保存到指定目录，并测量存储速度。
    checkpoint: 要保存的 checkpoint 数据。
    save_dir: 保存目录。
    """
    start_time = time.time()
    os.makedirs(save_dir, exist_ok=True)
    save_path = os.path.join(save_dir, "checkpoint.pt")
    torch.save(checkpoint, save_path)
    end_time = time.time()

    # 计算存储速度 (MB/s)
    file_size_mb = os.path.getsize(save_path) / (1024 * 1024)
    duration = end_time - start_time
    speed_mb_per_sec = file_size_mb / duration

    return speed_mb_per_sec

def main():
    parser = argparse.ArgumentParser(description="Test storage speed for checkpoint saving.")
    parser.add_argument("--size_mb", type=int, default=100, help="Size of the dummy checkpoint in MB (default: 100MB)")
    parser.add_argument("--save_dir", type=str, default="/3fs/stage/checkpoint", help="Directory to save the checkpoint (default: /3fs/stage/checkpoint)")
    args = parser.parse_args()

    # 生成模拟的 checkpoint 文件
    checkpoint = generate_dummy_checkpoint(args.size_mb)

    # 数与存储速度
    speed = save_checkpoint(checkpoint, args.save_dir)
    print(f"Storage speed: {speed:.2f} MB/s")

if __name__ == "__main__":
    main()

Save it as torch_test.py.

Install dependencies

1
pip install torch -i  https://pypi.mirrors.ustc.edu.cn/simple

Test

1
python3 torch_test.py --size_mb 2000 --save_dir /3fs/stage/checkpoint

1
Storage speed: 246.50 MB/s

4. Model Load Speed Test

The model tested here is DeepSeek R1.

1
2
3
4
5
6
ls /3fs/stage/model

chkp.py                            model-00084-of-000163.safetensors
config.json                        model-00085-of-000163.safetensors
configuration_deepseek.py          model-00086-of-000163.safetensors
...

The model is about 687 GB in size.

1
2
3
du -sh /3fs/stage/model

687G    /3fs/stage/model

Run the test command on two H100s at the same time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
python3 -u -m sglang.launch_server \
    --model-path /3fs/stage/model \
    --quantization fp8 \
    --tp 16 \
    --chunked-prefill-size 4096 \
    --dist-init-addr ${MASTER_ADDR}:${PORT} \
    --nnodes ${NNODES} \
    --node-rank ${NODE_RANK} \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000

Two tests were run in total; the first load took 10 minutes and the second took 12 minutes.

You can see that when two Pods load the model at the same time, the total bandwidth is about 1~1.2 GBps.

5. 4K Random Read/Write

4K read/write determines small-file read/write performance and has a significant impact on scenarios such as builds. This test was carried out on a different 3FS cluster with 5 storage nodes.

Host disk 4K random read/write

1
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=100m -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Write_Testing -filename=/data1/fiofile

1
  WRITE: bw=97.5MiB/s (102MB/s), 97.5MiB/s-97.5MiB/s (102MB/s-102MB/s), io=100MiB (105MB), run=1026-1026msec

1
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=100m -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Read_Testing -filename=/data1/fiofile

1
   READ: bw=1408MiB/s (1477MB/s), 1408MiB/s-1408MiB/s (1477MB/s-1477MB/s), io=100MiB (105MB), run=71-71msec

3FS 4K random read/write

1
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=100m -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Write_Testing -filename=/mnt/3fs/fiofile

1
  WRITE: bw=746KiB/s (764kB/s), 746KiB/s-746KiB/s (764kB/s-764kB/s), io=99.9MiB (105MB), run=137123-137123msec

1
fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=100m -numjobs=1 -runtime=1000 -group_reporting --allow_mounted_write=1 -name=Rand_Read_Testing -filename=/mnt/3fs/fiofile

1
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=100MiB (105MB), run=789-789msec

Summary

Operation	Read/Write Rate	Device
4K read	1408 MiB/s	Disk
4K write	97.5 MiB/s	Disk
4K read	127 MiB/s	3FS
4K write	746 KiB/s	3FS