Please enable Javascript to view the contents

ext4、xfs、zfs、btrfs 在 RAID6 下的性能测试

 ·  ☕ 16 分钟

1. 单盘

1.1 创建文件系统

  • 清理磁盘
1

wipefs -a /dev/nvme0n1
  • 格式化磁盘
1

mkfs.xfs /dev/nvme0n1
  • 挂载磁盘
1
2

mkdir -p /mnt/nvme0n1-test
mount /dev/nvme0n1 /mnt/nvme0n1-test

1.2 测试性能

  • 4k 随机写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/nvme0n1-test/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][w=970MiB/s][w=248k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1909379
  write: IOPS=235k, BW=919MiB/s (964MB/s)(108GiB/120001msec); 0 zone resets
    slat (usec): min=2, max=979550, avg=17.07, stdev=525.74
    clat (nsec): min=944, max=60651k, avg=1060067.20, stdev=832980.74
     lat (usec): min=20, max=980100, avg=1077.38, stdev=987.11
    clat percentiles (usec):
     |  1.00th=[  343],  5.00th=[  519], 10.00th=[  619], 20.00th=[  668],
     | 30.00th=[  709], 40.00th=[  791], 50.00th=[  922], 60.00th=[ 1057],
     | 70.00th=[ 1205], 80.00th=[ 1401], 90.00th=[ 1614], 95.00th=[ 1745],
     | 99.00th=[ 2376], 99.50th=[ 5014], 99.90th=[11076], 99.95th=[14222],
     | 99.99th=[27395]
   bw (  KiB/s): min=52536, max=1151376, per=100.00%, avg=946201.45, stdev=17317.50, samples=1904
   iops        : min=13134, max=287844, avg=236550.35, stdev=4329.38, samples=1904
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.14%
  lat (usec)   : 500=4.28%, 750=31.30%, 1000=20.26%
  lat (msec)   : 2=42.17%, 4=1.22%, 10=0.42%, 20=0.15%, 50=0.02%
  lat (msec)   : 100=0.01%
  cpu          : usr=10.95%, sys=34.24%, ctx=19512693, majf=0, minf=12967
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,28241228,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=919MiB/s (964MB/s), 919MiB/s-919MiB/s (964MB/s-964MB/s), io=108GiB (116GB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=7795/28219461, merge=0/0, ticks=894/21461568, in_queue=21462462, util=99.74%
  • 4k 随机读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/nvme0n1-test/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=1618MiB/s][r=414k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1912551
  read: IOPS=413k, BW=1614MiB/s (1692MB/s)(189GiB/120003msec)
    slat (nsec): min=1918, max=23825k, avg=7052.22, stdev=10803.55
    clat (usec): min=16, max=35556, avg=611.04, stdev=554.43
     lat (usec): min=32, max=35568, avg=618.23, stdev=554.54
    clat percentiles (usec):
     |  1.00th=[   94],  5.00th=[  113], 10.00th=[  141], 20.00th=[  202],
     | 30.00th=[  273], 40.00th=[  351], 50.00th=[  449], 60.00th=[  562],
     | 70.00th=[  717], 80.00th=[  930], 90.00th=[ 1287], 95.00th=[ 1663],
     | 99.00th=[ 2507], 99.50th=[ 2900], 99.90th=[ 3916], 99.95th=[ 4490],
     | 99.99th=[10945]
   bw (  MiB/s): min= 1492, max= 1693, per=100.00%, avg=1615.94, stdev= 3.55, samples=1912
   iops        : min=382186, max=433586, avg=413680.06, stdev=907.93, samples=1912
  lat (usec)   : 20=0.01%, 50=0.01%, 100=2.24%, 250=24.86%, 500=27.75%
  lat (usec)   : 750=17.09%, 1000=10.61%
  lat (msec)   : 2=14.84%, 4=2.53%, 10=0.08%, 20=0.01%, 50=0.01%
  cpu          : usr=16.08%, sys=44.37%, ctx=17601497, majf=0, minf=5047
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=49583832,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1614MiB/s (1692MB/s), 1614MiB/s-1614MiB/s (1692MB/s-1692MB/s), io=189GiB (203GB), run=120003-120003msec

Disk stats (read/write):
  nvme0n1: ios=49546789/2, merge=0/0, ticks=29364649/0, in_queue=29364648, util=100.00%
  • 4k 随机读写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/nvme0n1-test/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=233MiB/s,w=100MiB/s][r=59.7k,w=25.7k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1917796
  read: IOPS=59.9k, BW=234MiB/s (245MB/s)(27.4GiB/120001msec)
    slat (usec): min=2, max=12515, avg= 7.15, stdev= 9.01
    clat (nsec): min=771, max=24656k, avg=112575.37, stdev=61675.06
     lat (usec): min=15, max=24667, avg=119.89, stdev=62.41
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   90], 20.00th=[   92],
     | 30.00th=[   93], 40.00th=[   95], 50.00th=[  103], 60.00th=[  109],
     | 70.00th=[  110], 80.00th=[  130], 90.00th=[  157], 95.00th=[  174],
     | 99.00th=[  237], 99.50th=[  265], 99.90th=[  326], 99.95th=[  367],
     | 99.99th=[ 1598]
   bw (  KiB/s): min=225784, max=257424, per=100.00%, avg=239785.33, stdev=532.51, samples=1912
   iops        : min=56446, max=64356, avg=59946.35, stdev=133.12, samples=1912
  write: IOPS=25.7k, BW=100MiB/s (105MB/s)(11.8GiB/120001msec); 0 zone resets
    slat (usec): min=2, max=8215, avg= 7.65, stdev=10.00
    clat (nsec): min=638, max=24452k, avg=18600.38, stdev=60415.22
     lat (usec): min=11, max=24468, avg=26.41, stdev=61.41
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   18], 60.00th=[   18],
     | 70.00th=[   19], 80.00th=[   20], 90.00th=[   22], 95.00th=[   25],
     | 99.00th=[   37], 99.50th=[   48], 99.90th=[   95], 99.95th=[  124],
     | 99.99th=[  457]
   bw (  KiB/s): min=93880, max=112776, per=100.00%, avg=102798.74, stdev=411.55, samples=1912
   iops        : min=23470, max=28194, avg=25699.67, stdev=102.88, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.05%, 20=25.62%, 50=4.20%
  lat (usec)   : 100=34.21%, 250=35.41%, 500=0.47%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=4.46%, sys=11.58%, ctx=10291639, majf=0, minf=5604
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7186561,3080936,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=27.4GiB (29.4GB), run=120001-120001msec
  WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=11.8GiB (12.6GB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=7178343/3077415, merge=0/0, ticks=795079/40898, in_queue=835977, util=99.99%
  • 4M 顺序写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/nvme0n1-test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=976MiB/s][w=244 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1903815
  write: IOPS=245, BW=983MiB/s (1031MB/s)(57.6GiB/60031msec); 0 zone resets
    slat (usec): min=407, max=8882, avg=968.11, stdev=212.68
    clat (usec): min=4385, max=81306, avg=31579.35, stdev=5785.37
     lat (usec): min=4905, max=82386, avg=32547.98, stdev=5778.14
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[24511], 10.00th=[28181], 20.00th=[29754],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[30802], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32375], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[52167], 99.50th=[56361], 99.90th=[64750], 99.95th=[66847],
     | 99.99th=[72877]
   bw (  KiB/s): min=950272, max=1048576, per=100.00%, avg=1007720.76, stdev=20941.03, samples=119
   iops        : min=  232, max=  256, avg=246.03, stdev= 5.11, samples=119
  lat (msec)   : 10=0.03%, 20=3.25%, 50=95.16%, 100=1.56%
  cpu          : usr=9.38%, sys=14.64%, ctx=14556, majf=0, minf=2374
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14752,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=983MiB/s (1031MB/s), 983MiB/s-983MiB/s (1031MB/s-1031MB/s), io=57.6GiB (61.9GB), run=60031-60031msec

Disk stats (read/write):
  nvme0n1: ios=0/481501, merge=0/7, ticks=0/14164438, in_queue=14164438, util=99.91%
  • 4M 顺序读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/nvme0n1-test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2939MiB/s][r=734 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1907021
  read: IOPS=740, BW=2962MiB/s (3106MB/s)(174GiB/60007msec)
    slat (usec): min=204, max=3437, avg=580.49, stdev=137.23
    clat (usec): min=1369, max=31000, avg=10217.58, stdev=1029.34
     lat (usec): min=1994, max=31582, avg=10798.45, stdev=1016.62
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9503],
     | 30.00th=[ 9765], 40.00th=[10028], 50.00th=[10159], 60.00th=[10421],
     | 70.00th=[10683], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
     | 99.00th=[12518], 99.50th=[13042], 99.90th=[17171], 99.95th=[19530],
     | 99.99th=[29492]
   bw (  MiB/s): min= 2832, max= 3192, per=100.00%, avg=2964.73, stdev=44.43, samples=120
   iops        : min=  708, max=  798, avg=740.97, stdev=11.11, samples=120
  lat (msec)   : 2=0.01%, 4=0.14%, 10=38.39%, 20=61.42%, 50=0.05%
  cpu          : usr=0.35%, sys=44.25%, ctx=40915, majf=0, minf=11582
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=44441,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=2962MiB/s (3106MB/s), 2962MiB/s-2962MiB/s (3106MB/s-3106MB/s), io=174GiB (186GB), run=60007-60007msec

Disk stats (read/write):
  nvme0n1: ios=1505398/10, merge=9/6, ticks=6272903/1, in_queue=6272903, util=99.88%

1.3 清理

1
2

umount /mnt/nvme0n1-test
wipefs -a /dev/nvme0n1

2. ext4

2.1 创建文件系统

  • 创建 RAID6
1

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
  • 格式化 RAID6
1

mkfs.ext4 -F /dev/md0
  • 挂载 RAID6
1
2

mkdir -p /mnt/raid6-ext4
mount /dev/md0 /mnt/raid6-ext4

2.2 测试性能

  • 4k 随机写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-ext4/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=158MiB/s][w=40.5k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1921021
  write: IOPS=34.5k, BW=135MiB/s (141MB/s)(15.8GiB/120005msec); 0 zone resets
    slat (usec): min=3, max=54445, avg=87.25, stdev=330.51
    clat (usec): min=237, max=119183, avg=7338.96, stdev=2989.31
     lat (usec): min=302, max=123776, avg=7426.39, stdev=3053.93
    clat percentiles (usec):
     |  1.00th=[ 5407],  5.00th=[ 5932], 10.00th=[ 6128], 20.00th=[ 6390],
     | 30.00th=[ 6587], 40.00th=[ 6783], 50.00th=[ 6980], 60.00th=[ 7177],
     | 70.00th=[ 7439], 80.00th=[ 7832], 90.00th=[ 8455], 95.00th=[ 8979],
     | 99.00th=[12387], 99.50th=[20055], 99.90th=[55313], 99.95th=[70779],
     | 99.99th=[93848]
   bw (  KiB/s): min=94648, max=166944, per=100.00%, avg=137850.87, stdev=1858.05, samples=1912
   iops        : min=23662, max=41736, avg=34462.36, stdev=464.52, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.04%, 4=0.21%, 10=97.50%, 20=1.75%, 50=0.37%
  lat (msec)   : 100=0.12%, 250=0.01%
  cpu          : usr=1.87%, sys=19.38%, ctx=6049293, majf=0, minf=4725
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4135584,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=15.8GiB (16.9GB), run=120005-120005msec

Disk stats (read/write):
    md0: ios=0/4411647, merge=0/0, ticks=0/31116932, in_queue=31116932, util=100.00%, aggrios=1053022/3151757, aggrmerge=457303/223146, aggrticks=361746/801096, aggrin_queue=1162842, aggrutil=99.97%
  nvme0n1: ios=1051796/3163738, merge=456583/241374, ticks=333187/790180, in_queue=1123367, util=99.97%
  nvme3n1: ios=1054364/3139558, merge=457404/205297, ticks=374904/795141, in_queue=1170044, util=99.96%
  nvme2n1: ios=1052708/3163485, merge=456232/241732, ticks=393093/833331, in_queue=1226424, util=99.97%
  nvme1n1: ios=1053220/3140249, merge=458995/204183, ticks=345800/785734, in_queue=1131534, util=99.96%
  • 4k 随机读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-ext4/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3781MiB/s][r=968k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1928379
  read: IOPS=980k, BW=3827MiB/s (4013MB/s)(448GiB/120002msec)
    slat (usec): min=2, max=22326, avg= 5.36, stdev= 9.55
    clat (usec): min=5, max=28319, avg=255.26, stdev=368.93
     lat (usec): min=30, max=28328, avg=260.69, stdev=369.08
    clat percentiles (usec):
     |  1.00th=[   88],  5.00th=[   95], 10.00th=[  104], 20.00th=[  117],
     | 30.00th=[  133], 40.00th=[  149], 50.00th=[  169], 60.00th=[  192],
     | 70.00th=[  225], 80.00th=[  273], 90.00th=[  371], 95.00th=[  553],
     | 99.00th=[ 2311], 99.50th=[ 2737], 99.90th=[ 3294], 99.95th=[ 3490],
     | 99.99th=[ 7701]
   bw (  MiB/s): min= 3432, max= 4118, per=100.00%, avg=3828.99, stdev=12.81, samples=1912
   iops        : min=878655, max=1054446, avg=980221.39, stdev=3279.92, samples=1912
  lat (usec)   : 10=0.01%, 50=0.01%, 100=7.77%, 250=68.39%, 500=18.08%
  lat (usec)   : 750=1.99%, 1000=0.67%
  lat (msec)   : 2=1.64%, 4=1.44%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=16.79%, sys=69.27%, ctx=12007646, majf=0, minf=2518
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=117557831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3827MiB/s (4013MB/s), 3827MiB/s-3827MiB/s (4013MB/s-4013MB/s), io=448GiB (482GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=117436690/43522, merge=0/0, ticks=26505304/140684, in_queue=26645988, util=100.00%, aggrios=29450147/46750, aggrmerge=1865650/1395475, aggrticks=6687322/11001, aggrin_queue=6698324, aggrutil=100.00%
  nvme0n1: ios=29452652/53570, merge=1885324/1622929, ticks=7386033/12314, in_queue=7398348, util=100.00%
  nvme3n1: ios=29449336/39735, merge=1843379/1168037, ticks=6111870/9676, in_queue=6121545, util=100.00%
  nvme2n1: ios=29449103/53885, merge=1885349/1622856, ticks=7134533/12459, in_queue=7146992, util=100.00%
  nvme1n1: ios=29449500/39811, merge=1848551/1168078, ticks=6116853/9557, in_queue=6126411, util=100.00%
  • 4k 随机读写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-ext4/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=78.6MiB/s,w=33.2MiB/s][r=20.1k,w=8493 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1931989
  read: IOPS=20.0k, BW=78.1MiB/s (81.9MB/s)(9369MiB/120001msec)
    slat (usec): min=2, max=14600, avg= 9.47, stdev=20.28
    clat (nsec): min=707, max=14475k, avg=114057.25, stdev=53174.77
     lat (usec): min=16, max=14603, avg=123.66, stdev=57.02
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   91], 20.00th=[   93],
     | 30.00th=[   94], 40.00th=[   96], 50.00th=[  101], 60.00th=[  109],
     | 70.00th=[  111], 80.00th=[  124], 90.00th=[  157], 95.00th=[  178],
     | 99.00th=[  273], 99.50th=[  322], 99.90th=[  461], 99.95th=[  537],
     | 99.99th=[ 1139]
   bw (  KiB/s): min=50080, max=106709, per=100.00%, avg=79991.88, stdev=1347.29, samples=1912
   iops        : min=12520, max=26677, avg=19997.68, stdev=336.82, samples=1912
  write: IOPS=8576, BW=33.5MiB/s (35.1MB/s)(4020MiB/120001msec); 0 zone resets
    slat (usec): min=3, max=47580, avg=26.45, stdev=522.23
    clat (usec): min=2, max=42389, avg=611.31, stdev=1909.48
     lat (usec): min=39, max=51574, avg=637.88, stdev=1997.61
    clat percentiles (usec):
     |  1.00th=[  122],  5.00th=[  131], 10.00th=[  141], 20.00th=[  163],
     | 30.00th=[  206], 40.00th=[  255], 50.00th=[  297], 60.00th=[  347],
     | 70.00th=[  400], 80.00th=[  469], 90.00th=[  578], 95.00th=[  979],
     | 99.00th=[10421], 99.50th=[14877], 99.90th=[25297], 99.95th=[28443],
     | 99.99th=[32637]
   bw (  KiB/s): min=22024, max=45560, per=100.00%, avg=34322.94, stdev=565.57, samples=1912
   iops        : min= 5506, max=11390, avg=8580.33, stdev=141.38, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=34.73%, 250=45.84%, 500=14.47%, 750=3.19%, 1000=0.27%
  lat (msec)   : 2=0.31%, 4=0.48%, 10=0.39%, 20=0.25%, 50=0.07%
  cpu          : usr=1.75%, sys=5.21%, ctx=3542124, majf=0, minf=5452
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2398355,1029155,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.1MiB/s (81.9MB/s), 78.1MiB/s-78.1MiB/s (81.9MB/s-81.9MB/s), io=9369MiB (9824MB), run=120001-120001msec
  WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=4020MiB (4215MB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=2397573/1070445, merge=0/0, ticks=277788/754272, in_queue=1032060, util=99.05%, aggrios=1025481/860471, aggrmerge=5183988/2579731, aggrticks=187967/90796, aggrin_queue=278764, aggrutil=99.92%
  nvme0n1: ios=1024560/882495, merge=5193905/3211896, ticks=193732/91576, in_queue=285309, util=99.92%
  nvme3n1: ios=1027163/837628, merge=5172424/1947710, ticks=186838/86010, in_queue=272848, util=99.82%
  nvme2n1: ios=1025610/882143, merge=5193934/3212044, ticks=190461/103978, in_queue=294439, util=99.86%
  nvme1n1: ios=1024594/839619, merge=5175691/1947276, ticks=180838/81623, in_queue=262461, util=99.82%
  • 4M 顺序写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-ext4

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1057MiB/s][w=264 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1934647
  write: IOPS=252, BW=1011MiB/s (1061MB/s)(59.3GiB/60009msec); 0 zone resets
    slat (usec): min=696, max=29051, avg=3947.48, stdev=2894.86
    clat (usec): min=7854, max=65807, avg=27672.29, stdev=4476.28
     lat (usec): min=9388, max=73138, avg=31620.58, stdev=5097.91
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[22414], 10.00th=[23200], 20.00th=[24249],
     | 30.00th=[25035], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28705], 80.00th=[30540], 90.00th=[33817], 95.00th=[36439],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[52691], 99.95th=[55313],
     | 99.99th=[65799]
   bw (  KiB/s): min=663552, max=1171456, per=99.96%, avg=1035287.98, stdev=98841.97, samples=119
   iops        : min=  162, max=  286, avg=252.71, stdev=24.14, samples=119
  lat (msec)   : 10=0.01%, 20=0.30%, 50=99.48%, 100=0.21%
  cpu          : usr=8.03%, sys=34.33%, ctx=47249, majf=0, minf=5932
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15174,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=1011MiB/s (1061MB/s), 1011MiB/s-1011MiB/s (1061MB/s-1061MB/s), io=59.3GiB (63.6GB), run=60009-60009msec

Disk stats (read/write):
    md0: ios=0/485973, merge=0/0, ticks=0/4879688, in_queue=4879688, util=100.00%, aggrios=8249/255090, aggrmerge=160430/7571355, aggrticks=7477/85380, aggrin_queue=92857, aggrutil=99.98%
  nvme0n1: ios=8911/255924, merge=177042/7582089, ticks=8065/100379, in_queue=108443, util=99.98%
  nvme3n1: ios=8025/253914, merge=154163/7558632, ticks=6573/82685, in_queue=89257, util=99.50%
  nvme2n1: ios=7725/255928, merge=147521/7581260, ticks=7598/80680, in_queue=88278, util=99.53%
  nvme1n1: ios=8335/254597, merge=162995/7563440, ticks=7675/77777, in_queue=85452, util=99.56%
  • 4M 顺序读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-ext4

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=9000MiB/s][r=2250 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1937785
  read: IOPS=2270, BW=9082MiB/s (9523MB/s)(532GiB/60004msec)
    slat (usec): min=238, max=2234, avg=414.98, stdev=62.30
    clat (usec): min=417, max=18489, avg=3106.55, stdev=1225.58
     lat (usec): min=718, max=18927, avg=3521.79, stdev=1234.17
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[ 1221], 10.00th=[ 1631], 20.00th=[ 2114],
     | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3294],
     | 70.00th=[ 3589], 80.00th=[ 3916], 90.00th=[ 4424], 95.00th=[ 5014],
     | 99.00th=[ 7308], 99.50th=[ 8160], 99.90th=[10028], 99.95th=[10814],
     | 99.99th=[13566]
   bw (  MiB/s): min= 7760, max=10016, per=100.00%, avg=9084.13, stdev=370.03, samples=119
   iops        : min= 1940, max= 2504, avg=2270.90, stdev=92.50, samples=119
  lat (usec)   : 500=0.03%, 750=1.15%, 1000=1.89%
  lat (msec)   : 2=12.70%, 4=66.07%, 10=18.05%, 20=0.11%
  cpu          : usr=0.59%, sys=94.39%, ctx=9051, majf=0, minf=15159
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=136233,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=9082MiB/s (9523MB/s), 9082MiB/s-9082MiB/s (9523MB/s-9523MB/s), io=532GiB (571GB), run=60004-60004msec

Disk stats (read/write):
    md0: ios=5172987/6, merge=0/0, ticks=3219800/0, in_queue=3219800, util=100.00%, aggrios=1879189/13211, aggrmerge=1371752/405448, aggrticks=1002496/2476, aggrin_queue=1004973, aggrutil=100.00%
  nvme0n1: ios=1892021/17620, merge=1371802/540532, ticks=1065268/3056, in_queue=1068323, util=100.00%
  nvme3n1: ios=1908987/8824, merge=1371770/270360, ticks=872630/1670, in_queue=874301, util=100.00%
  nvme2n1: ios=1908990/17589, merge=1371735/540633, ticks=1083970/3423, in_queue=1087393, util=100.00%
  nvme1n1: ios=1806761/8814, merge=1371703/270270, ticks=988118/1757, in_queue=989876, util=100.00%

2.3 清理

1
2

umount /mnt/raid6-ext4
mdadm --stop /dev/md0

3. xfs

3.1 创建文件系统

  • 创建 RAID6
1

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
  • 格式化 RAID6
1

mkfs.xfs -f /dev/md0
  • 挂载 RAID6
1
2

mkdir -p /mnt/raid6-xfs
mount /dev/md0 /mnt/raid6-xfs

3.2 测试性能

  • 4k 随机写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-xfs/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=123MiB/s][w=31.4k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1940340
  write: IOPS=29.6k, BW=116MiB/s (121MB/s)(13.6GiB/120007msec); 0 zone resets
    slat (usec): min=3, max=26502, avg=20.73, stdev=136.15
    clat (usec): min=152, max=54282, avg=8622.32, stdev=1428.85
     lat (usec): min=165, max=54297, avg=8643.29, stdev=1431.81
    clat percentiles (usec):
     |  1.00th=[ 7046],  5.00th=[ 7504], 10.00th=[ 7635], 20.00th=[ 7898],
     | 30.00th=[ 8029], 40.00th=[ 8160], 50.00th=[ 8356], 60.00th=[ 8586],
     | 70.00th=[ 8717], 80.00th=[ 9110], 90.00th=[ 9765], 95.00th=[10421],
     | 99.00th=[13304], 99.50th=[15270], 99.90th=[25035], 99.95th=[28443],
     | 99.99th=[41157]
   bw (  KiB/s): min=94072, max=137312, per=100.00%, avg=118473.02, stdev=775.95, samples=1912
   iops        : min=23518, max=34328, avg=29617.89, stdev=194.00, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.05%, 10=92.11%, 20=7.58%, 50=0.24%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.93%, sys=6.29%, ctx=4986888, majf=0, minf=3586
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3553476,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=116MiB/s (121MB/s), 116MiB/s-116MiB/s (121MB/s-121MB/s), io=13.6GiB (14.6GB), run=120007-120007msec

Disk stats (read/write):
    md0: ios=0/3613480, merge=0/0, ticks=0/30817252, in_queue=30817252, util=99.92%, aggrios=917507/2696600, aggrmerge=641848/356680, aggrticks=320494/752749, aggrin_queue=1073244, aggrutil=100.00%
  nvme0n1: ios=916162/2697378, merge=642093/357094, ticks=295017/755565, in_queue=1050583, util=100.00%
  nvme3n1: ios=916874/2697845, merge=641434/356363, ticks=330966/747980, in_queue=1078947, util=100.00%
  nvme2n1: ios=919207/2695099, merge=641483/356687, ticks=349058/753163, in_queue=1102222, util=100.00%
  nvme1n1: ios=917787/2696081, merge=642385/356576, ticks=306936/754291, in_queue=1061227, util=100.00%
  • 4k 随机读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-xfs/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=3774MiB/s][r=966k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1952661
  read: IOPS=1056k, BW=4124MiB/s (4324MB/s)(483GiB/120001msec)
    slat (usec): min=2, max=24986, avg= 6.17, stdev= 9.24
    clat (usec): min=28, max=25523, avg=235.60, stdev=115.81
     lat (usec): min=31, max=25535, avg=241.85, stdev=116.33
    clat percentiles (usec):
     |  1.00th=[  108],  5.00th=[  129], 10.00th=[  143], 20.00th=[  161],
     | 30.00th=[  176], 40.00th=[  192], 50.00th=[  208], 60.00th=[  231],
     | 70.00th=[  258], 80.00th=[  297], 90.00th=[  363], 95.00th=[  433],
     | 99.00th=[  594], 99.50th=[  660], 99.90th=[  824], 99.95th=[  906],
     | 99.99th=[ 1401]
   bw (  MiB/s): min= 3618, max= 4466, per=100.00%, avg=4125.69, stdev=16.94, samples=1912
   iops        : min=926327, max=1143355, avg=1056175.34, stdev=4335.63, samples=1912
  lat (usec)   : 50=0.01%, 100=0.45%, 250=67.12%, 500=29.93%, 750=2.30%
  lat (usec)   : 1000=0.18%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=17.52%, sys=81.94%, ctx=571508, majf=0, minf=2912
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=126680444,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=4124MiB/s (4324MB/s), 4124MiB/s-4124MiB/s (4324MB/s-4324MB/s), io=483GiB (519GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=126511644/13, merge=0/0, ticks=21021424/0, in_queue=21021424, util=100.00%, aggrios=31779374/89, aggrmerge=3381181/20, aggrticks=5311275/4, aggrin_queue=5311279, aggrutil=100.00%
  nvme0n1: ios=31779520/91, merge=3381169/26, ticks=5308492/4, in_queue=5308495, util=100.00%
  nvme3n1: ios=31781887/91, merge=3381175/26, ticks=5317892/5, in_queue=5317896, util=100.00%
  nvme2n1: ios=31779391/88, merge=3381176/26, ticks=5316364/5, in_queue=5316369, util=100.00%
  nvme1n1: ios=31776698/87, merge=3381206/2, ticks=5302352/4, in_queue=5302356, util=100.00%
  • 4k 随机读写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-xfs/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=30.4MiB/s,w=12.9MiB/s][r=7782,w=3306 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1955381
  read: IOPS=20.5k, BW=80.0MiB/s (83.9MB/s)(9605MiB/120004msec)
    slat (usec): min=3, max=27865, avg= 9.68, stdev=44.57
    clat (nsec): min=915, max=21420k, avg=105529.98, stdev=53965.56
     lat (usec): min=18, max=31988, avg=115.33, stdev=71.25
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   91], 20.00th=[   92],
     | 30.00th=[   93], 40.00th=[   95], 50.00th=[   96], 60.00th=[  104],
     | 70.00th=[  110], 80.00th=[  111], 90.00th=[  120], 95.00th=[  153],
     | 99.00th=[  210], 99.50th=[  249], 99.90th=[  347], 99.95th=[  429],
     | 99.99th=[  840]
   bw (  KiB/s): min=21200, max=174135, per=100.00%, avg=82182.77, stdev=5469.20, samples=1912
   iops        : min= 5300, max=43532, avg=20545.33, stdev=1367.29, samples=1912
  write: IOPS=8793, BW=34.3MiB/s (36.0MB/s)(4122MiB/120004msec); 0 zone resets
    slat (usec): min=3, max=41563, avg=57.07, stdev=743.53
    clat (nsec): min=888, max=55891k, avg=577249.86, stdev=1959976.86
     lat (usec): min=33, max=55906, avg=634.44, stdev=2107.59
    clat percentiles (usec):
     |  1.00th=[  120],  5.00th=[  128], 10.00th=[  137], 20.00th=[  149],
     | 30.00th=[  161], 40.00th=[  176], 50.00th=[  194], 60.00th=[  219],
     | 70.00th=[  260], 80.00th=[  367], 90.00th=[  478], 95.00th=[  775],
     | 99.00th=[11469], 99.50th=[15533], 99.90th=[21627], 99.95th=[22938],
     | 99.99th=[27919]
   bw (  KiB/s): min= 8872, max=74673, per=100.00%, avg=35269.38, stdev=2348.71, samples=1912
   iops        : min= 2218, max=18667, avg=8816.96, stdev=587.17, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=40.08%, 250=50.04%, 500=7.29%, 750=1.04%, 1000=0.17%
  lat (msec)   : 2=0.15%, 4=0.23%, 10=0.61%, 20=0.32%, 50=0.05%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.70%, sys=5.58%, ctx=3937367, majf=0, minf=2155
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2458810,1055200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=80.0MiB/s (83.9MB/s), 80.0MiB/s-80.0MiB/s (83.9MB/s-83.9MB/s), io=9605MiB (10.1GB), run=120004-120004msec
  WRITE: bw=34.3MiB/s (36.0MB/s), 34.3MiB/s-34.3MiB/s (36.0MB/s-36.0MB/s), io=4122MiB (4322MB), run=120004-120004msec

Disk stats (read/write):
    md0: ios=2458285/1055004, merge=0/0, ticks=254740/643324, in_queue=898064, util=100.00%, aggrios=1078700/791604, aggrmerge=6173324/44, aggrticks=212008/68723, aggrin_queue=280731, aggrutil=98.10%
  nvme0n1: ios=1079069/791504, merge=6173339/30, ticks=207819/64418, in_queue=272237, util=98.06%
  nvme3n1: ios=1077577/792216, merge=6173310/59, ticks=215384/69736, in_queue=285119, util=98.00%
  nvme2n1: ios=1077832/792186, merge=6173310/59, ticks=214590/73950, in_queue=288540, util=98.10%
  nvme1n1: ios=1080322/790511, merge=6173338/31, ticks=210242/66788, in_queue=277029, util=98.07%
  • 4M 顺序写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-xfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=964MiB/s][w=241 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1957297
  write: IOPS=221, BW=886MiB/s (929MB/s)(51.9GiB/60014msec); 0 zone resets
    slat (usec): min=777, max=28964, avg=4508.44, stdev=3157.21
    clat (usec): min=4967, max=81188, avg=31602.09, stdev=4772.63
     lat (usec): min=15779, max=83917, avg=36111.25, stdev=5350.46
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25297], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30802], 60.00th=[32113],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[47973], 99.90th=[56361], 99.95th=[61604],
     | 99.99th=[67634]
   bw (  KiB/s): min=761856, max=1048576, per=99.89%, avg=906108.73, stdev=73554.26, samples=119
   iops        : min=  186, max=  256, avg=221.14, stdev=17.96, samples=119
  lat (msec)   : 10=0.01%, 20=0.05%, 50=99.61%, 100=0.34%
  cpu          : usr=8.66%, sys=35.80%, ctx=42541, majf=0, minf=71654
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13291,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=51.9GiB (55.7GB), run=60014-60014msec

Disk stats (read/write):
    md0: ios=0/424172, merge=0/0, ticks=0/4428772, in_queue=4428772, util=99.97%, aggrios=8809/220705, aggrmerge=183902/6590883, aggrticks=9283/67789, aggrin_queue=77072, aggrutil=99.97%
  nvme0n1: ios=8438/221027, merge=175305/6592858, ticks=9150/85042, in_queue=94190, util=99.97%
  nvme3n1: ios=8844/220420, merge=183811/6589689, ticks=9286/63809, in_queue=73094, util=99.23%
  nvme2n1: ios=8946/220571, merge=187729/6588710, ticks=9383/61551, in_queue=70935, util=99.29%
  nvme1n1: ios=9011/220804, merge=188765/6592276, ticks=9313/60755, in_queue=70069, util=99.34%
  • 4M 顺序读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-xfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=9590MiB/s][r=2397 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1959857
  read: IOPS=2384, BW=9539MiB/s (10.0GB/s)(559GiB/60001msec)
    slat (usec): min=227, max=3364, avg=410.08, stdev=56.82
    clat (usec): min=397, max=15775, avg=2942.87, stdev=690.51
     lat (usec): min=741, max=19141, avg=3353.22, stdev=703.77
    clat percentiles (usec):
     |  1.00th=[  881],  5.00th=[ 2409], 10.00th=[ 2704], 20.00th=[ 2737],
     | 30.00th=[ 2769], 40.00th=[ 2802], 50.00th=[ 2835], 60.00th=[ 2868],
     | 70.00th=[ 2900], 80.00th=[ 2966], 90.00th=[ 3359], 95.00th=[ 4080],
     | 99.00th=[ 5932], 99.50th=[ 6783], 99.90th=[ 8586], 99.95th=[ 9110],
     | 99.99th=[10683]
   bw (  MiB/s): min= 8079, max=10256, per=100.00%, avg=9542.85, stdev=380.62, samples=119
   iops        : min= 2019, max= 2564, avg=2385.66, stdev=95.20, samples=119
  lat (usec)   : 500=0.02%, 750=0.57%, 1000=0.61%
  lat (msec)   : 2=1.83%, 4=91.57%, 10=5.39%, 20=0.02%
  cpu          : usr=0.63%, sys=97.74%, ctx=2739, majf=0, minf=8216
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=143082,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=9539MiB/s (10.0GB/s), 9539MiB/s-9539MiB/s (10.0GB/s-10.0GB/s), io=559GiB (600GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=5447134/5, merge=0/0, ticks=2711068/0, in_queue=2711068, util=100.00%, aggrios=1961115/42, aggrmerge=1873436/9, aggrticks=867493/1, aggrin_queue=867495, aggrutil=100.00%
  nvme0n1: ios=1977580/42, merge=1876309/5, ticks=852352/2, in_queue=852354, util=100.00%
  nvme3n1: ios=1953626/43, merge=1876454/12, ticks=832879/2, in_queue=832881, util=100.00%
  nvme2n1: ios=1977399/42, merge=1864556/10, ticks=912517/1, in_queue=912519, util=100.00%
  nvme1n1: ios=1935855/42, merge=1876425/9, ticks=872226/2, in_queue=872228, util=100.00%

3.3 清理

1
2

umount /mnt/raid6-xfs
mdadm --stop /dev/md0

4. zfs

4.1 创建文件系统

  • 创建 raidz2 池
1

zpool create -o ashift=12 -f tank raidz2 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
  • 挂载存储
1

zfs create -o mountpoint=/mnt/raid6-zfs tank/data

4.2 测试性能

  • 4k 随机写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-zfs/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=90.9MiB/s][w=23.3k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1963969
  write: IOPS=23.3k, BW=91.1MiB/s (95.5MB/s)(10.7GiB/120001msec); 0 zone resets
    slat (usec): min=6, max=78665, avg=340.44, stdev=302.79
    clat (usec): min=2, max=115643, avg=10640.06, stdev=3639.21
     lat (usec): min=170, max=115673, avg=10980.75, stdev=3741.09
    clat percentiles (usec):
     |  1.00th=[ 3163],  5.00th=[ 5669], 10.00th=[ 6456], 20.00th=[ 7570],
     | 30.00th=[ 8455], 40.00th=[ 9634], 50.00th=[10683], 60.00th=[11600],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[14222], 95.00th=[16188],
     | 99.00th=[21365], 99.50th=[23200], 99.90th=[31589], 99.95th=[38011],
     | 99.99th=[63177]
   bw (  KiB/s): min=65471, max=166791, per=100.00%, avg=93304.09, stdev=1815.69, samples=1912
   iops        : min=16367, max=41697, avg=23325.98, stdev=453.93, samples=1912
  lat (usec)   : 4=0.01%, 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=0.56%, 4=0.57%, 10=42.33%, 20=54.68%, 50=1.80%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.87%, sys=19.72%, ctx=1994732, majf=0, minf=11071
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2797218,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=91.1MiB/s (95.5MB/s), 91.1MiB/s-91.1MiB/s (95.5MB/s-95.5MB/s), io=10.7GiB (11.5GB), run=120001-120001msec
  • 4k 随机读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-zfs/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 7 (f=0): [f(2),_(1),f(5)][100.0%][r=496MiB/s][r=127k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=167510
  read: IOPS=135k, BW=527MiB/s (553MB/s)(61.8GiB/120001msec)
    slat (usec): min=3, max=43573, avg=58.05, stdev=79.10
    clat (nsec): min=1544, max=48675k, avg=1838908.19, stdev=499838.11
     lat (usec): min=47, max=48751, avg=1897.03, stdev=509.35
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1696],
     | 30.00th=[ 1745], 40.00th=[ 1778], 50.00th=[ 1827], 60.00th=[ 1860],
     | 70.00th=[ 1893], 80.00th=[ 1926], 90.00th=[ 2008], 95.00th=[ 2073],
     | 99.00th=[ 2376], 99.50th=[ 2704], 99.90th=[ 7373], 99.95th=[12256],
     | 99.99th=[23987]
   bw (  KiB/s): min=461232, max=652588, per=100.00%, avg=539742.12, stdev=3565.08, samples=1912
   iops        : min=115308, max=163146, avg=134935.13, stdev=891.27, samples=1912
  lat (usec)   : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=90.09%, 4=9.73%, 10=0.11%, 20=0.05%, 50=0.02%
  cpu          : usr=2.04%, sys=96.95%, ctx=21818, majf=0, minf=5155
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=16187984,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=527MiB/s (553MB/s), 527MiB/s-527MiB/s (553MB/s-553MB/s), io=61.8GiB (66.3GB), run=120001-120001msec
  • 4k 随机读写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-zfs/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=204MiB/s,w=86.9MiB/s][r=52.2k,w=22.2k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=327904
  read: IOPS=54.6k, BW=213MiB/s (224MB/s)(25.0GiB/120001msec)
    slat (usec): min=2, max=57794, avg=55.87, stdev=80.66
    clat (nsec): min=391, max=2255.5k, avg=637.85, stdev=1870.43
     lat (usec): min=3, max=57800, avg=56.61, stdev=80.77
    clat percentiles (nsec):
     |  1.00th=[  422],  5.00th=[  446], 10.00th=[  462], 20.00th=[  482],
     | 30.00th=[  502], 40.00th=[  540], 50.00th=[  572], 60.00th=[  604],
     | 70.00th=[  652], 80.00th=[  716], 90.00th=[  852], 95.00th=[ 1032],
     | 99.00th=[ 1480], 99.50th=[ 1688], 99.90th=[ 2832], 99.95th=[ 5600],
     | 99.99th=[18816]
   bw (  KiB/s): min=186400, max=299796, per=100.00%, avg=218550.90, stdev=1930.64, samples=1912
   iops        : min=46600, max=74949, avg=54637.47, stdev=482.66, samples=1912
  write: IOPS=23.4k, BW=91.5MiB/s (95.9MB/s)(10.7GiB/120001msec); 0 zone resets
    slat (usec): min=7, max=59822, avg=205.29, stdev=193.74
    clat (nsec): min=410, max=1320.0k, avg=854.01, stdev=2124.52
     lat (usec): min=8, max=59828, avg=206.37, stdev=194.13
    clat percentiles (nsec):
     |  1.00th=[  498],  5.00th=[  532], 10.00th=[  564], 20.00th=[  612],
     | 30.00th=[  644], 40.00th=[  684], 50.00th=[  724], 60.00th=[  780],
     | 70.00th=[  852], 80.00th=[  980], 90.00th=[ 1240], 95.00th=[ 1544],
     | 99.00th=[ 2448], 99.50th=[ 2864], 99.90th=[ 4704], 99.95th=[ 8640],
     | 99.99th=[22400]
   bw (  KiB/s): min=79988, max=129866, per=100.00%, avg=93686.46, stdev=813.81, samples=1912
   iops        : min=19997, max=32466, avg=23421.44, stdev=203.45, samples=1912
  lat (nsec)   : 500=20.53%, 750=54.78%, 1000=15.12%
  lat (usec)   : 2=8.80%, 4=0.68%, 10=0.06%, 20=0.02%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.76%, sys=57.53%, ctx=1716012, majf=0, minf=4905
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6554099,2809430,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=213MiB/s (224MB/s), 213MiB/s-213MiB/s (224MB/s-224MB/s), io=25.0GiB (26.8GB), run=120001-120001msec
  WRITE: bw=91.5MiB/s (95.9MB/s), 91.5MiB/s-91.5MiB/s (95.9MB/s-95.9MB/s), io=10.7GiB (11.5GB), run=120001-120001msec
  • 4M 顺序写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-zfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2000MiB/s][w=500 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2579345
  write: IOPS=575, BW=2303MiB/s (2415MB/s)(135GiB/60003msec); 0 zone resets
    slat (usec): min=1293, max=13604, avg=1732.63, stdev=256.27
    clat (usec): min=6, max=44204, avg=12155.63, stdev=1445.06
     lat (usec): min=2129, max=49405, avg=13888.73, stdev=1623.53
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[10814], 10.00th=[11076], 20.00th=[11338],
     | 30.00th=[11469], 40.00th=[11600], 50.00th=[11731], 60.00th=[11863],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13960], 95.00th=[15008],
     | 99.00th=[17171], 99.50th=[18220], 99.90th=[23725], 99.95th=[25297],
     | 99.99th=[40109]
   bw (  MiB/s): min= 1704, max= 2448, per=100.00%, avg=2305.14, stdev=140.38, samples=119
   iops        : min=  426, max=  612, avg=576.29, stdev=35.09, samples=119
  lat (usec)   : 10=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=99.75%, 50=0.23%
  cpu          : usr=14.70%, sys=84.39%, ctx=2041, majf=0, minf=24658
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,34554,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=2303MiB/s (2415MB/s), 2303MiB/s-2303MiB/s (2415MB/s-2415MB/s), io=135GiB (145GB), run=60003-60003msec
  • 4M 顺序读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-zfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=1846MiB/s][r=461 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=3990788
  read: IOPS=472, BW=1891MiB/s (1983MB/s)(111GiB/60002msec)
    slat (usec): min=1771, max=3993, avg=2111.42, stdev=116.84
    clat (usec): min=3, max=27204, avg=14804.46, stdev=675.49
     lat (usec): min=2042, max=31201, avg=16916.39, stdev=759.75
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[13960], 10.00th=[14091], 20.00th=[14222],
     | 30.00th=[14353], 40.00th=[14615], 50.00th=[14746], 60.00th=[14877],
     | 70.00th=[15139], 80.00th=[15270], 90.00th=[15533], 95.00th=[15795],
     | 99.00th=[17171], 99.50th=[17433], 99.90th=[18220], 99.95th=[19006],
     | 99.99th=[23725]
   bw (  MiB/s): min= 1784, max= 1984, per=100.00%, avg=1891.43, stdev=44.34, samples=119
   iops        : min=  446, max=  496, avg=472.86, stdev=11.09, samples=119
  lat (usec)   : 4=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=99.94%, 50=0.04%
  cpu          : usr=0.16%, sys=99.82%, ctx=170, majf=0, minf=16427
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=28366,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=1891MiB/s (1983MB/s), 1891MiB/s-1891MiB/s (1983MB/s-1983MB/s), io=111GiB (119GB), run=60002-60002msec

4.3 清理

1

zpool destroy -f tank

5. btrfs

5.1 创建文件系统

  • 创建 RAID6
1

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
  • 格式化 RAID6
1

mkfs.btrfs -f /dev/md0
  • 挂载 RAID6
1
2

mkdir -p /mnt/raid6-btrfs
mount /dev/md0 /mnt/raid6-btrfs

5.2 测试性能

  • 4k 随机写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-btrfs/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=194MiB/s][w=49.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=3994479
  write: IOPS=42.2k, BW=165MiB/s (173MB/s)(19.3GiB/120005msec); 0 zone resets
    slat (usec): min=10, max=32078, avg=96.53, stdev=261.42
    clat (usec): min=398, max=46208, avg=5962.46, stdev=1939.07
     lat (usec): min=423, max=46279, avg=6059.17, stdev=1970.81
    clat percentiles (usec):
     |  1.00th=[ 3621],  5.00th=[ 4080], 10.00th=[ 4490], 20.00th=[ 5211],
     | 30.00th=[ 5473], 40.00th=[ 5669], 50.00th=[ 5800], 60.00th=[ 5932],
     | 70.00th=[ 6128], 80.00th=[ 6390], 90.00th=[ 6980], 95.00th=[ 8029],
     | 99.00th=[11076], 99.50th=[15795], 99.90th=[34341], 99.95th=[37487],
     | 99.99th=[41681]
   bw (  KiB/s): min=108776, max=206328, per=99.97%, avg=168894.49, stdev=2337.59, samples=1912
   iops        : min=27194, max=51582, avg=42223.59, stdev=584.40, samples=1912
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=4.00%, 10=94.46%, 20=1.14%, 50=0.36%
  cpu          : usr=2.00%, sys=42.74%, ctx=3123061, majf=0, minf=29643
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5068581,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=19.3GiB (20.8GB), run=120005-120005msec
  • 4k 随机读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-btrfs/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=1365MiB/s][r=349k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=3998132
  read: IOPS=407k, BW=1588MiB/s (1665MB/s)(186GiB/120001msec)
    slat (usec): min=5, max=29813, avg=18.14, stdev=18.26
    clat (usec): min=84, max=31354, avg=610.73, stdev=185.39
     lat (usec): min=98, max=31366, avg=628.97, stdev=187.08
    clat percentiles (usec):
     |  1.00th=[  437],  5.00th=[  482], 10.00th=[  502], 20.00th=[  529],
     | 30.00th=[  562], 40.00th=[  586], 50.00th=[  603], 60.00th=[  627],
     | 70.00th=[  644], 80.00th=[  668], 90.00th=[  709], 95.00th=[  758],
     | 99.00th=[  889], 99.50th=[  988], 99.90th=[ 1401], 99.95th=[ 1991],
     | 99.99th=[ 9110]
   bw (  MiB/s): min= 1327, max= 2076, per=100.00%, avg=1589.36, stdev=18.22, samples=1912
   iops        : min=339942, max=531600, avg=406875.78, stdev=4663.81, samples=1912
  lat (usec)   : 100=0.01%, 250=0.01%, 500=9.69%, 750=84.78%, 1000=5.06%
  lat (msec)   : 2=0.41%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=6.83%, sys=92.61%, ctx=56035, majf=0, minf=4323
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=48790319,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1588MiB/s (1665MB/s), 1588MiB/s-1588MiB/s (1665MB/s-1665MB/s), io=186GiB (200GB), run=120001-120001msec
  • 4k 随机读写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-btrfs/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=31.9MiB/s,w=13.8MiB/s][r=8174,w=3527 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=4001506
  read: IOPS=16.6k, BW=64.9MiB/s (68.0MB/s)(7786MiB/120002msec)
    slat (usec): min=7, max=8169, avg=23.86, stdev=10.77
    clat (nsec): min=838, max=27750k, avg=121119.69, stdev=86765.10
     lat (usec): min=37, max=27786, avg=145.17, stdev=87.95
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[   99], 10.00th=[  101], 20.00th=[  103],
     | 30.00th=[  104], 40.00th=[  106], 50.00th=[  111], 60.00th=[  118],
     | 70.00th=[  121], 80.00th=[  125], 90.00th=[  157], 95.00th=[  178],
     | 99.00th=[  255], 99.50th=[  302], 99.90th=[  457], 99.95th=[  594],
     | 99.99th=[ 3326]
   bw (  KiB/s): min=15120, max=157704, per=100.00%, avg=66654.69, stdev=3811.96, samples=1912
   iops        : min= 3780, max=39426, avg=16663.67, stdev=952.99, samples=1912
  write: IOPS=7128, BW=27.8MiB/s (29.2MB/s)(3341MiB/120002msec); 0 zone resets
    slat (usec): min=11, max=46642, avg=89.32, stdev=907.10
    clat (nsec): min=860, max=47070k, avg=685854.72, stdev=2479658.53
     lat (usec): min=40, max=48153, avg=775.36, stdev=2660.01
    clat percentiles (usec):
     |  1.00th=[   33],  5.00th=[   39], 10.00th=[   49], 20.00th=[   72],
     | 30.00th=[  109], 40.00th=[  145], 50.00th=[  176], 60.00th=[  212],
     | 70.00th=[  277], 80.00th=[  375], 90.00th=[  537], 95.00th=[ 2343],
     | 99.00th=[13960], 99.50th=[19268], 99.90th=[26608], 99.95th=[28967],
     | 99.99th=[35914]
   bw (  KiB/s): min= 6720, max=68872, per=100.00%, avg=28606.03, stdev=1627.74, samples=1912
   iops        : min= 1680, max=17218, avg=7151.51, stdev=406.93, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=3.24%
  lat (usec)   : 100=11.10%, 250=74.91%, 500=7.19%, 750=1.34%, 1000=0.21%
  lat (msec)   : 2=0.42%, 4=0.36%, 10=0.69%, 20=0.39%, 50=0.13%
  cpu          : usr=1.64%, sys=10.42%, ctx=3173151, majf=0, minf=8419
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1993297,855419,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=64.9MiB/s (68.0MB/s), 64.9MiB/s-64.9MiB/s (68.0MB/s-68.0MB/s), io=7786MiB (8165MB), run=120002-120002msec
  WRITE: bw=27.8MiB/s (29.2MB/s), 27.8MiB/s-27.8MiB/s (29.2MB/s-29.2MB/s), io=3341MiB (3504MB), run=120002-120002msec
  • 4M 顺序写
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1041MiB/s][w=260 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=4004221
  write: IOPS=245, BW=982MiB/s (1030MB/s)(57.6GiB/60010msec); 0 zone resets
    slat (usec): min=1387, max=51095, avg=4064.23, stdev=2197.78
    clat (usec): min=7280, max=75665, avg=28498.74, stdev=4611.73
     lat (usec): min=9874, max=87549, avg=32563.74, stdev=5080.03
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[23725], 10.00th=[24249], 20.00th=[25297],
     | 30.00th=[26084], 40.00th=[26608], 50.00th=[27395], 60.00th=[28181],
     | 70.00th=[29230], 80.00th=[31065], 90.00th=[34341], 95.00th=[36439],
     | 99.00th=[44827], 99.50th=[50070], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[73925]
   bw (  KiB/s): min=704512, max=1114112, per=99.94%, avg=1005275.43, stdev=88912.09, samples=119
   iops        : min=  172, max=  272, avg=245.43, stdev=21.71, samples=119
  lat (msec)   : 10=0.01%, 20=0.04%, 50=99.47%, 100=0.47%
  cpu          : usr=7.55%, sys=56.10%, ctx=28659, majf=0, minf=7210
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14737,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=982MiB/s (1030MB/s), 982MiB/s-982MiB/s (1030MB/s-1030MB/s), io=57.6GiB (61.8GB), run=60010-60010msec
  • 4M 顺序读
1
2
3
4

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=837MiB/s][r=209 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=4009283
  read: IOPS=216, BW=866MiB/s (908MB/s)(50.8GiB/60004msec)
    slat (usec): min=3892, max=15785, avg=4611.47, stdev=405.68
    clat (usec): min=397, max=54320, avg=32314.95, stdev=2261.59
     lat (usec): min=4833, max=63086, avg=36927.46, stdev=2531.49
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[29492], 10.00th=[29754], 20.00th=[30540],
     | 30.00th=[31065], 40.00th=[31589], 50.00th=[32113], 60.00th=[32637],
     | 70.00th=[33162], 80.00th=[33817], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[40109], 99.50th=[42730], 99.90th=[46924], 99.95th=[50070],
     | 99.99th=[51643]
   bw (  KiB/s): min=712704, max=983040, per=99.99%, avg=886786.23, stdev=44116.29, samples=119
   iops        : min=  174, max=  240, avg=216.50, stdev=10.77, samples=119
  lat (usec)   : 500=0.01%
  lat (msec)   : 10=0.02%, 20=0.02%, 50=99.91%, 100=0.05%
  cpu          : usr=0.17%, sys=99.77%, ctx=260, majf=0, minf=8214
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=12992,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=866MiB/s (908MB/s), 866MiB/s-866MiB/s (908MB/s-908MB/s), io=50.8GiB (54.5GB), run=60004-60004msec

5.3 清理

1

umount /mnt/raid6-btrfs

6. 对比

测试结果（Latency 为 fio 报告的 clat avg，CPU 为 usr+sys）。在四盘位 RAID6 软阵列下，各文件系统性能对比。

6.1 4k - 随机写

文件系统进程数IOPSBW(MiB/s)Latency(us)CPU(%)
ext4834.5k135733921.3
xfs829.6k11686228.2
zfs823.3k91.11064020.6
btrfs842.2k165596244.7
单盘8235k919106045.2
理论 RAID6—58.8k230——
  • 4k 随机写时，btrfs 占用更多 CPU，性能最接近理论值

6.2 4k - 随机读

文件系统进程数IOPSBW(MiB/s)Latency(us)CPU(%)
ext48980k382725586.1
xfs81056k412423699.5
zfs8135k527183999.0
btrfs8407k158861199.4
单盘8413k161461160.5
理论 RAID6—1652k6456——
  • 4k 随机读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

6.3 4k - 随机读写

70% 随机读，30% 随机写。

文件系统进程数IOPS读IOPS写BW读(MiB/s)BW写(MiB/s)Latency读(us)Latency写(us)CPU(%)
ext4820.0k8.6k78.133.51146117.0
xfs820.5k8.8k80.034.31065777.3
zfs854.6k23.4k21391.50.640.8559.3
btrfs816.6k7.1k64.927.812168612.1
单盘859.9k25.7k23410011218.616.0
理论 RAID6—240k6.4k93625———
  • 4k 随机读写时，zfs 明显占用更多 CPU，性能明显优于其他文件系统

6.4 4M - 顺序写

文件系统进程数IOPSBW(MiB/s)Latency(ms)CPU(%)
ext41252101127.742.4
xfs122188631.644.5
zfs1575230312.299.1
btrfs124598228.563.7
单盘124598331.624.0
理论 RAID6—4901966——
  • 4M 顺序写时，zfs 占用更多 CPU，性能最接近理论值

6.5 4M - 顺序读

文件系统进程数IOPSBW(MiB/s)Latency(us/ms)CPU(%)
ext41227090823107us95.0
xfs1238495392943us98.4
zfs1472189114.8ms100
btrfs121686632.3ms99.9
单盘1740296210.2ms44.6
理论 RAID6—296011848——
  • 4M 顺序读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

7. 结论

RAID6 需要计算大量校验数据，占用更多 CPU。

  • 4k 随机写时，btrfs 占用更多 CPU，性能最接近理论值
  • 4k 随机读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈
  • 4k 随机读写时，zfs 明显占用更多 CPU，性能明显优于其他文件系统
  • 4M 顺序写时，zfs 占用更多 CPU，性能最接近理论值
  • 4M 顺序读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

选型建议：大文件顺序读、高吞吐读用 xfs 或 ext4；大文件顺序写优先 zfs；小文件随机 IO 用 ext4、btrfs；备份归档用 xfs、zfs。

微信公众号
作者
微信公众号

相关内容